January 26th, 2011

Making NFS suck in containters, part eleventy jillion.

A lawyer friend of mine describes her job as "being paid to stick her hands in other people's toilets". I am being paid to fix NFS. Right, gloves on.

The design of NFS is so utterly horrible that if I start describing what's wrong with it I'll be ranting for the rest of my work day, so let's just skip to the end: I'm starting with a read only NFSv3 UDP, skipping the port mapper and explicitly specifying the mountd and nfsd ports.

To export a simple NFSv3 filesystem from userspace (I.E. the sane way), first you need an exports file, with at least one line: An absolute path to a directory (with no whitespace in it), then a space, then a parenthetical with mount options. (Which options? Even though there is no sane way to ever export NFS directly to the internet without your box immediately being rooted, you have to go out of your way to _disable_ NFS's idea of "security" features.) So:</p>

/home/landley/test (no_root_squash,insecure)


There is a man page on this file format (man 5 exports). It's very long.

Let's launch a userspace nfs server, bound to loopback, with the "mount" and "nfs" protocols on port 9999. I'm using unfs3, as the least insane options I've found so far. (This is not an endorsement.)

unfsd -d -s -p -e $(pwd)/export -l 127.0.0.1 -m 9999 -n 9999


The -d (debug) option says not to background and write log messages to stdout. This lets us kill the server with ctrl-c when we need to relaunch it. (Note: don't forget to umount before you do this or your system goes all unstable and you get some fun hangs. The 2.6 kernel sucks much less than previous versions, specifically the "umount -f" option is your friend. And it taints the kernel!)

The -p option says not to fiddle with the portmapper. (Otherwise you need to run the server as root.) When we mount, we'll explicitly specify what ports to find the server on. (Why does it look for the portmapper on a standard port, to look up what other ports to connect to? See "design is utterly horrible".)

The -s (squash UIDs) option passes through the current UID to the client, and translates everything else to root. (It's also because we're not running the server as root, so it can't fiddle with multiple UIDs.)

The -e option says what to export. (No, you can't just list what to export on the command line, you instead have to feed it a config file listing the exports. I don't know why unfsd needs an absolue path to the export file, but it fails with an error message if you just name the darn file. See "least insane is not an endorsement".)

The -l says bind to a specific interface instead of all of the interfaces on your system. (Due to fundamental design flaws you can't securely expose NFS directly to the internet. We're not running the server as root, but we're still just binding to loopback anyway.)

The -n and -m options specify the port for the nfsd and mountd services. (Why are there two? See "utterly horrible design".)

Now in another directory, let's mount the sucker:

mkdir blah
sudo mount -t nfs -o ro,port=9999,mountport=9999,nolock,v3,udp \
  127.0.0.1:/home/landley/test blah
ls -l blah


We explicitly specify the port for nfsd and mountd (thus bypassing portmapd), disable the lock extensions (yet another server), tell it to use v3 of the protocol, and tell it to use the older (UDP datagram) transport. (The path we exported is the filesystem identifier, if there's a way to replace that I haven't found it yet.)

Mounting it read only hopefully makes some of the weird caching issues not bite on this pass, but I may need to come back and disable some of them explicitly.