NFS: Explanation & Insights

Network File System: a protocol that lets a Linux machine mount and use a directory stored on another computer as if it were local.

What It Is

NFS stands for Network File System, and the name is almost embarrassingly literal: it's a filesystem that lives across a network. One server holds the actual files on its actual disks, and other machines reach across the wire and mount that directory into their own tree. After that, the remote directory is just there — under /mnt/shared or /home or wherever you hung it — and your programs read and write it with the same ls, cp, and open() calls they'd use on a local disk. They never know the bytes are travelling over Ethernet. That transparency is the whole magic trick, and it's why NFS has survived, more or less unchanged in spirit, since 1984.

If you're coming from application development, the mental model is simpler than it sounds. You already know that a path like /var/log/syslog doesn't tell you which physical disk holds it — the filesystem layer hides that. NFS just stretches that same hiding a little further: the "disk" now happens to be a different computer. The kernel on the client intercepts every file operation under the mount point, packages it into a network request, ships it to the server, and hands you back the answer. From userspace, a read is a read.

There are two halves to any NFS setup, and keeping them straight will save you hours:

The server — the machine that owns the files and exports a directory, declaring "this folder is available over the network, to these clients, with these rules." Its export list lives in /etc/exports.
The client — the machine that mounts the export and uses it. It runs no special daemon beyond the kernel's NFS code; it just needs to know the server's address and the export path.

A single box can be both at once (export /data to others while mounting /backups from a third machine), but the roles never blur in any one direction: for a given share, somebody serves and somebody consumes.

Why It Matters

The reason NFS keeps showing up on servers is that shared state is hard, and NFS makes one slice of it easy. Web app running on three front-end machines, all needing the same uploaded user avatars? Put the avatars on an NFS server and mount it on all three — now there's one copy, one source of truth, and no syncing scripts to go wrong at 3 a.m. A render farm where forty nodes chew on the same project files? NFS. A /home directory that follows users to whichever machine they log into? That's the original 1980s use case, and it still works.

But NFS matters for a second, darker reason, and it's the one this page exists to nail: a sick NFS mount can take down a perfectly healthy server. Not the file server — the client. A machine with idle CPUs, free memory, and nothing wrong with its own disks can show a load average of 80 because of a file server that vanished across the network. That failure mode is so counterintuitive, and so common, that we'll spend a whole section on it below. If you take one thing from this page, take that.

So NFS earns its place — but it's a tool with a sharp edge, and the difference between "quietly perfect" and "the whole fleet is wedged" is a handful of mount options and a couple of /etc/exports decisions. Let's make them the right ones.

How It Works

Exports — the Server's Side of the Deal

Everything starts on the server with a single file: /etc/exports. Each line says which directory, to whom, and under what rules:

# /etc/exports
/srv/shared   10.0.0.0/24(rw,sync,no_subtree_check)
/srv/media    10.0.0.50(ro,sync,no_subtree_check)
/home         *.internal.example(rw,sync,root_squash)

The first field is the directory to share. The second is the client specification — a single IP, a CIDR range like 10.0.0.0/24, a hostname, or a wildcard like *.internal.example. Then, in parentheses with no space before the bracket (a classic first-time trap — a space changes the meaning entirely), come the options:

rw / ro — read-write or read-only. Default to ro and only open rw where you mean it.
sync / async — whether the server confirms writes only after they hit stable storage (sync, the safe default) or acknowledges early (async, faster but it can lie to the client about durability across a server crash).
no_subtree_check — disables a fragile old security check that caused more bugs than it ever prevented; the modern recommended setting.
root_squash — the important one, covered in its own breath below.

After editing /etc/exports, you don't restart anything heavy — you tell the server to re-read it:

exportfs -ra        # re-export everything, applying changes from /etc/exports
exportfs -v         # show what's currently exported and with which options

The exportfs command is the server's control panel for the export table. exportfs -ra is the one you'll type after every edit; forgetting it is why "but I added the line!" mounts still get refused.

root_squash — Why Root on the Client Isn't Root on the Server

This deserves its own moment because it's a real security boundary, not a footnote. By default, NFS applies root_squash: when a process running as root (UID 0) on a client touches a file on the export, the server quietly remaps that identity to a harmless unprivileged user (nobody, usually UID 65534). The thinking is sound — the file server has no way to know whether the client's root is your trusted admin or somebody who just rooted a compromised laptop on the same LAN. So it refuses to grant root's powers across the wire.

The dangerous-sounding inverse is no_root_squash, which tells the server "trust this client's root completely — let it act as root on my files." There are narrow, legitimate uses (diskless clients booting their entire OS over NFS), but for general file sharing it hands any client with root the keys to your server's data.

Danger

Never export with no_root_squash to a client you don't fully control. Anyone who gains root on that client — or anyone who can spoof its IP on an unsegmented network — can then create setuid-root binaries and rewrite any file on the export as root. Keep the default root_squash unless you have a diskless-boot reason not to, and even then, lock the export to one specific trusted address.

Discovering and Mounting an Export

From the client, you can ask a server what it offers with showmount:

showmount -e fileserver01      # list the exports a server advertises

Export list for fileserver01:
/srv/shared 10.0.0.0/24
/srv/media  10.0.0.50

Then you mount it. The source is written host:/path, and the command is an ordinary mount with a type of nfs:

mount -t nfs fileserver01:/srv/shared /mnt/shared

Now /mnt/shared on the client is /srv/shared on the server. List it, write to it, watch the bytes appear on the far end. To make it permanent, you add a line to /etc/fstab — and this is where one option matters more than any other:

fileserver01:/srv/shared  /mnt/shared  nfs  _netdev,noatime  0  0

The _netdev flag tells the boot process "this mount needs the network, so don't even try it until networking is up." Leave it off and a server that reboots while the file server is unreachable can hang during boot, waiting forever for a mount that can't possibly succeed yet. _netdev is cheap insurance against a 20-minute boot stall and an emergency console session, so put it on every NFS line in /etc/fstab, every time.

Pro Tip

Always add _netdev to NFS entries in /etc/fstab. It reorders the mount to run after the network is online, so a reboot during a file-server outage degrades gracefully (the mount is skipped or retried in the background) instead of wedging the whole boot. Pair it with noatime while you're there — there's rarely a reason to ship a write across the network just to record that you read a file.

NFSv3 vs NFSv4 — The One Distinction That Actually Matters

NFS comes in versions, and unlike most version bumps, this one changes how you reason about the protocol. You'll meet two in the wild: NFSv3 (1995, still everywhere) and NFSv4 (2000 onward, what you should be using on anything new).

The deep difference is state. NFSv3 is stateless by design: the server keeps no memory of which clients have which files open. Every request carries everything the server needs to answer it, standalone. That sounds elegant — and it made server crashes survivable, because a rebooted server had nothing to "forget" — but it has a cost. A stateless server can't, on its own, manage file locking, so NFSv3 bolts locking on through a separate protocol (NLM, the Network Lock Manager), running as its own daemon on its own port. It also leans on the old rpcbind/portmapper machinery to negotiate which ports the various pieces (mountd, lockd, statd) are listening on — and those ports float, by default, all over the place.

Which leads to the practical headache: NFSv3 is a nightmare to firewall. You don't open one port; you open a fistful, several of them dynamically assigned, and you fight rpcbind to pin them down. On a flat trusted LAN nobody cared. The moment a firewall sits between client and server, NFSv3 turns into a debugging session.

NFSv4 threw all of that out and started fresh:

It's stateful. The server tracks opens and locks itself, so locking is built into the protocol — no separate NLM daemon, no lockd, no statd.
It uses a single port: 2049. Mount, lock, read, write — everything rides one well-known TCP port. Firewalling becomes "allow 2049/tcp" and you're done.
Security is integrated. NFSv4 was designed with strong authentication in mind (Kerberos via RPCSEC_GSS), and it introduced a cleaner export model where everything hangs off a single pseudo-root rather than a scatter of independent exports.
It's the assumed default in modern Linux. Type mount -t nfs on a current distro and it'll negotiate the highest version both ends support, which today means v4.

	NFSv3	NFSv4
State	Stateless	Stateful
Locking	Separate protocol (NLM/`lockd`)	Built into the protocol
Ports	`rpcbind` + several, some dynamic	Single port 2049
Firewalling	Painful	One rule
Auth	AUTH_SYS (trust the UID) by default	Kerberos-capable, cleaner model
Year	1995	2000+

The verdict is not subtle: use NFSv4. The single-port design alone is worth it, the integrated locking removes a whole class of "the lock daemon died" mysteries, and the security model is the one written for a world that has firewalls in it. The only reason to touch v3 is a stubborn old appliance that speaks nothing else — and even then, treat it as a migration target, not a destination.

The Hung-Mount Trap — Load Average Through the Roof, CPU Asleep

Here's the war story this page was written for, and it's the kind of thing that gets quietly cited in incident reviews. You log into a server because monitoring screamed about a load average of 80. You run top, braced for a runaway process eating every core — and instead you find a box that's bored. CPUs near idle. Memory fine. No process burning cycles. And yet the load number sits up there at 80, 100, climbing, on a machine that by every CPU measure is asleep. What on earth is going on?

The answer is one of the most important things to understand about Linux, and NFS is the classic way to discover it: load average on Linux is not CPU usage. It counts processes that are runnable — but it also counts processes stuck in uninterruptible sleep, the state the tools call D. A process in D-state is blocked inside a kernel call, waiting on I/O that the kernel has promised will eventually complete, and it cannot be woken, paused, or even killed until that I/O finishes. It's not using the CPU at all. It's just waiting — and every one of those waiters adds 1 to the load.

Now picture a hard-mounted NFS share whose server just vanished — crashed, rebooted, lost its network, someone tripped over the cable. Every process that so much as stat()s a file under that mount point — your backup job, a ls you typed, a monitoring agent walking the tree, the shell's tab-completion — issues a network request to a server that will never answer. The kernel parks each one in D-state, waiting for a reply that isn't coming. One by one they pile up. Ten cron jobs touch the mount, that's load 10. A directory-walking script spawns fifty workers, that's load 60. The box does nothing — and reports a catastrophe.

And here's the part that turns a confused engineer into an angry one: kill -9 won't free them. SIGKILL is the signal that always works, the one nothing can catch or ignore — except a process in uninterruptible sleep isn't ignoring it. It simply cannot be scheduled to receive it until the kernel I/O it's blocked on returns. The whole point of the D-state is that the kernel is mid-operation with promises outstanding (buffers, locks, in-flight network state) and waking the process early would corrupt things. So the processes sit there, unkillable, until the file server comes back — at which point they all silently complete as if nothing happened — or until you reboot the client. You can watch them in top or with ps:

ps -eo pid,stat,wchan:32,comm | grep '^ *[0-9]* D'

The D in the STAT column is the smoking gun, and the wchan (wait channel) column usually names an NFS function, telling you exactly which dead well they're all drinking from.

hard vs soft vs intr — the Knob That Controls All This

Whether a vanished server merely pauses your work or wedges it forever comes down to one mount option:

hard (the default) — retry the request forever. The process blocks in D-state until the server returns, however long that takes. This is the right default for data safety: a write that's been acknowledged to the application must not be silently dropped just because the network blipped, and hard guarantees the operation eventually completes correctly. The price is exactly the trap above — when the server truly never comes back, the client waits truly forever.
soft — give up after a timeout and return an I/O error to the application. The process is freed; load doesn't climb. But now a transient network hiccup can hand your program a write error mid-operation, and applications that don't check every return code can corrupt data quietly. soft trades safety for liveness, and you only want it for read-only data where a failed read is harmless.
intr — historically, "let signals interrupt a hard mount," so a stuck process could be killed. On modern kernels (2.6.25+) intr is a no-op — the behaviour was reworked — and certain fatal signals can now break a hard mount in some cases. Don't rely on it; mentioned because you'll see it in old fstab lines and ancient tutorials.

So the rule of thumb: keep hard for anything you write to (it's the default for good reason — your data's integrity depends on it), and reach for soft only on read-only mounts where you'd rather see an error than a hung process. If you must run hard mounts of a server that might disappear, the real protections are operational: _netdev so boot doesn't hang, sane timeo/retrans tuning so the client notices the outage quickly, and monitoring that tells you the file server died before forty D-state processes do.

Warning

Resist the urge to "fix" hung-mount load by switching every mount to soft. On a read-write mount, soft converts "my process pauses until the server returns" into "my process gets a write error and may leave a half-written file." For data you care about, a paused process is the safe failure — it resumes cleanly when the server comes back. Use soft only where every access is read-only.

Stale File Handles — ESTALE

The other NFS-specific gremlin announces itself with a message that looks like a typo: Stale file handle, error code ESTALE. Understanding it requires one more piece of how NFS works under the hood.

When a client opens a file on an export, the server doesn't hand back a path — it hands back a small, opaque token called a file handle that encodes (roughly) the filesystem and the file's inode number. The client clings to that handle and uses it for every subsequent read and write, never re-resolving the path. This is deliberate and efficient: the server doesn't have to walk /srv/shared/projects/active/report.txt from the top on every single operation; it just looks up the inode the handle points at.

The handle goes stale when the thing it points at stops existing on the server side while the client still holds the handle open. The two classic causes:

The file (or a directory above it) was deleted or recreated on the server — by another client, or directly on the server. The inode number the handle encodes is now either gone or reused for something else. Next time the client uses the handle: ESTALE.
The export itself changed underneath the mount — the server's filesystem was unmounted and remounted, a backup restore swapped the underlying inodes, or exportfs -r rebuilt things in a way that invalidated the handles. The path is still /srv/shared, but the inodes the clients memorised are no longer what's there.

To the application it's baffling: the file is right there, ls on the server shows it, but the client insists the handle is stale. That's because the client is asking about a specific inode it remembers, not the name you see. The fix is to make the client re-resolve: often just cd out of and back into the directory, or close and reopen the file; when it's stuck across a whole mount, unmount and remount the share (umount /mnt/shared && mount /mnt/shared). Persistent ESTALE across a mount usually means the server's export was rebuilt or its underlying filesystem was swapped — go look at what changed there.

Note

ESTALE is the protocol working as designed, not corruption. It's the price of the file-handle optimisation that makes NFS fast — the client trades "always re-check the path" for "remember the inode," and a stale handle is what happens when that bet loses. If you see it constantly, something on the server is churning inodes under the clients' feet (restores, rsync-with---inplace-gone-wrong, or two servers fighting over the same export).

Cheat Sheet

# --- On the server: define and apply exports ---
cat /etc/exports                       # the export table you edit by hand
exportfs -v                            # show what's exported right now, with options
exportfs -ra                           # re-read /etc/exports and apply (run after every edit)
exportfs -u client:/srv/shared         # un-export one entry without editing the file

# --- On the client: discover and mount ---
showmount -e fileserver01              # list exports a server advertises
mount -t nfs fileserver01:/srv/shared /mnt/shared            # mount (negotiates v4 by default)
mount -t nfs -o vers=4.2 host:/data /mnt/data                # pin a specific version
mount -t nfs -o ro,soft,_netdev host:/media /mnt/media       # read-only, fail fast, network-aware

# --- Persist across reboots (always _netdev) ---
# /etc/fstab:
#   fileserver01:/srv/shared  /mnt/shared  nfs  _netdev,noatime  0  0

# --- See what's mounted and how ---
mount -t nfs4                          # list current NFS mounts and their options
findmnt -t nfs4                        # same, in a clean tree

# --- Diagnose a hung mount ---
ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/'   # processes stuck in uninterruptible sleep
nfsstat -c                             # client-side NFS stats (retransmits hint at a sick server)
umount -f -l /mnt/shared               # force + lazy unmount a dead mount (last resort)

Gotchas

Things that catch people, roughly in order of how much head-scratching they cause:

A space before the parenthesis in /etc/exports changes everything. /srv (rw) exports /srv read-only to the world and read-write to a host literally named (rw). Write it tight: /srv 10.0.0.0/24(rw). No space, ever.
You edited /etc/exports but the client still can't mount. You forgot exportfs -ra. The file is just a definition; the running export table only updates when you tell it to re-read.
Boot hangs after a file server goes down. A missing _netdev on the fstab line. The mount runs before networking and blocks forever. Add _netdev to every NFS entry.
UIDs don't line up across machines. With the default AUTH_SYS, NFS trusts the client's numeric UID/GID — so user alice (UID 1001) on the client gets whatever UID 1001 owns on the server, even if that's bob. Keep your user databases in sync (LDAP, or NFSv4's idmapd), or you'll get files owned by the wrong person and permission errors that make no sense.
Load average of 80 on an idle box. A hard-mounted share to a dead server, processes piling up in D-state. Not a CPU problem — a network/storage problem. See the whole section above; it's the single most-misdiagnosed NFS symptom.
kill -9 does nothing to the stuck processes. They're in uninterruptible sleep waiting on the dead server; SIGKILL can't reach them until the I/O returns. Revive the server or reboot the client.
Stale file handle on a file that's clearly there. The inode the client memorised changed on the server. cd out and back, or remount; if it persists, something on the server rebuilt the export.
async on the export quietly risks your data. It makes writes faster by acknowledging them before they're durable — fine for scratch, a silent data-loss risk for anything you'd miss after a server crash. Default to sync.

History and Philosophy

NFS was born at Sun Microsystems in 1984, and it carried Sun's whole worldview in its bones: "the network is the computer." In an office full of diskless Sun workstations, your files didn't live on your machine — they lived on a server, and NFS made them appear on whichever workstation you happened to sit down at. That /home-follows-you experience, utterly ordinary now, was close to magic then, and NFS made it cheap enough to deploy everywhere. The protocol was published openly (RFC 1094 for v2), which is a big part of why it spread to every Unix and why it's still here forty years later when most of its contemporaries are museum pieces.

The original design leaned hard on statelessness, and that wasn't laziness — it was a genuinely clever answer to a hard problem. If the server keeps no per-client state, then a server crash and reboot is invisible to clients: there's nothing to reconstruct, no sessions to recover, the requests just start succeeding again. The cost, as we saw, was that anything genuinely stateful — locking, above all — had to be bolted on the side, and the bolt-ons (NLM, statd, the rpcbind dance) became NFSv3's most annoying baggage. NFSv4, two decades later, made peace with state: it accepted that tracking opens and locks in the protocol is worth it, folded everything onto a single port, and in doing so quietly admitted that the firewall-shaped world had won. It's a nice arc — a protocol confident enough in its original idea to abandon it once the idea's assumptions stopped holding.

Philosophically, NFS sits at a fault line that runs through all of distributed computing: the dream of transparency versus the stubbornness of physics. The promise is that a remote file behaves exactly like a local one — same syscalls, same semantics, no special handling. And most of the time it delivers. But the network is not a disk; it has partitions, latencies, and outright disappearances that a local disk simply doesn't, and the D-state hang is what it looks like when that leaky abstraction finally leaks. NFS is, in the end, an honest teacher: use it for years and it works invisibly, and then one day a file server reboots and it hands you a free, vivid lesson in what load average really counts. Worth the tuition.