How can you not be romantic about UNIX domain sockets?
Earlier this summer, I gave a very technical talk at the DEFCON 34 convention (search up “Rage Against the Sandbox”) and my demo crashed on stage. I played it off cool and just re-launched the demo and it all worked pretty nicely. Honestly, what kept my nervousness up wasn’t that but the fact I was about to run a 1-day LPE exploit with 36% success rate on stage (but it’s Vegas, so I figured: what the hell, and I came on top of the house with a success on first try on stage). Coming back home, I began working on the final touches of the project I presented before releasing the code publicly and it made me want to dig deeper into the crashing issue.
The investigation
I noticed this crash occurs only after the device is freshly booted. It never happens the second time on the same boot. It just always happened, which is weird because it means it’s not something affected by randomness (such as ASLR or multi-thread race conditions).
Let’s get a little familiar with the details. My program is a VM running an SSH server inside an iOS application. The basic idea of the program is that iOS does not permit multi-processing in applications and the VM implements multi-processing semantics by creating threads into the same process with duplicated resources and memory. The VM itself implements TTY in userspace by creating a pair of UNIX domain sockets connected to each other and performing pre-processing on the data sent into each side (when master sends Ctrl+C, slave gets a SIGINT, etc). Since the TTY spec defines only 1 managing terminal per-process we cannot rely on the iOS kernel’s implementation because we won’t be able to have multiple SSH sessions through the same iOS application process. The crash on stage was at the userspace TTY implementation, and specifically at this snippet:
// tvm.c
/**
* for a given VM-managed file-descriptor,
* pull out the associated TTY object.
*/
static struct tty *
tty_for_file_locked(struct file *file, int *out_ttymode) {
// ...
struct tty *tt = (struct tty *)file->f_data;
VERIFY(tt->t_refcnt);
// ...
struct stat st;
fstat(file->f_rfd, &st);
// ...
if (tt->t_mfd_ino == st.st_ino) {
*out_ttymode = TTM_MASTER;
return tt;
}
VERIFY(tt->t_sfd_ino == st.st_ino); // sanity only XXX: CRASH HERE!
*out_ttymode = TTM_SLAVE;
return tt;
}Since file-descriptors can be dup()ed around, I’m keeping the original inode numbers of the UNIX domain socket ends and I distinguish the TTY mode of the file by it (master/slave). The crash is at VERIFY(tt->t_sfd_ino == st.st_ino). That sanity check fails and the VM panics. It seemed to me that there’s some kind of memory corruption out there, because it would make no sense for a file-descriptor to change its associated inode.
The crash itself happens somewhere along the way of the initial SSH connection that sets up the master/slave end of the terminal session for that connection. The SSH server is dropbear, a popular open-source embedded SSH implementation, so I assume there’s nothing wrong with the codebase but probably my implementation.
After debugging the returned inode numbers from fstat() inside the VM, something seemed very weird. It didn’t look like a corruption anymore. It appears like an fstat() call on one end of the socket yielded a different inode on the second call to fstat() on the same socket (second call is in the code snippet shown above).
The bug
After debugging a little more trying to convince myself something is wrong in my code (perhaps the socket I fstat() changed??), I came to the point where nothing made sense to me. Code seemed good, crash was deterministic on the first run of the demo after reboot, it has to be something else. So, I did the obvious thing I’d been avoiding: Just open the kernel code bro. And so, I did:
// /bsd/kern/uipc_usrreq.c
static int
uipc_sense(struct socket *so, void *ub, int isstat64)
{
struct unpcb *unp = sotounpcb(so);
struct socket *so2;
blksize_t blksize;
if (unp == 0) {
return EINVAL;
}
blksize = so->so_snd.sb_hiwat;
if (so->so_type == SOCK_STREAM && unp->unp_conn != 0) {
so2 = unp->unp_conn->unp_socket;
blksize += so2->so_rcv.sb_cc;
}
if (unp->unp_ino == 0) {
unp->unp_ino = unp_ino++;
}
if (isstat64 != 0) {
struct stat64 *sb64;
sb64 = (struct stat64 *)ub;
sb64->st_blksize = blksize;
sb64->st_dev = NODEV;
sb64->st_ino = (ino64_t)unp->unp_ino;
}
// ...
return 0;
}That snippet of uipc_sense() implements the logic that pulls out the socket’s inode into the st_ino field of struct stat. You can see the implementation lazily assigns an inode on the first call to stat on the target socket (unp->unp_ino == 0) from some global variable named unp_ino.
Can you spot the bug?
If you still want to find it yourself, you better stop reading because the next sentence will reveal the answer. The bug has nothing to do with race conditions over the global variable as evident by the fact it is deterministic but rather the global variable usage itself - it should be ++unp_ino rather than unp_ino++. The unp_ino global variable is initialized to zero (as most global data should be), and the check (unp->unp_ino == 0) assumes unp_ino means an uninitialized inode field on the socket. But the first ever call to uipc_sense() on the system will cause unp_ino++ to yield 0 (where ++unp_ino will yield 1). This causes the first ever socket to have fstat() called on it (which receives inode 0), to change its inode on the second stat to something else because the assumption is 0 is an uninitialized inode.
First of all, quite a wow. While not a very interesting bug from a security standpoint, having found a proper kernel logic bug that breaks userspace on the DEFCON stage was a shocker to realize. At that time, I did not know how long this bug had been in the operating system, but it kept me wondering, how many programs are impacted by this bug? Also, how the hell is my demo the first ever process on the iPhone to call fstat() on a UNIX domain socket? Is there some system service on every iPhone out there that might be holding an erroneous inode to a UNIX domain socket?
Second, the fix is super easy on the kernel side. But since my code needs to support versions backwards, I probably want to add an Apple-specific check to re-call fstat() if I get 0 as the inode number. And that really hit me with the curiosity: If all versions of iOS prior to whichever would fix this bug are impacted, what introduced this bug and how far back does it go?
The history
The most obvious way to figure out how old the bug is: just pull up the earliest commit of XNU in GitHub and read uipc_sense(). This is XNU 123.5 released on March 24, 2001 with Mac OS X 10.0:
// /bsd/kern/uipc_usrreq.c
static int
uipc_sense(struct socket *so, struct stat *sb)
{
struct unpcb *unp = sotounpcb(so);
struct socket *so2;
if (unp == 0)
return EINVAL;
sb->st_blksize = so->so_snd.sb_hiwat;
if (so->so_type == SOCK_STREAM && unp->unp_conn != 0) {
so2 = unp->unp_conn->unp_socket;
sb->st_blksize += so2->so_rcv.sb_cc;
}
sb->st_dev = NODEV;
if (unp->unp_ino == 0)
unp->unp_ino = unp_ino++;
sb->st_ino = unp->unp_ino;
return (0);
}Womp womp, simpler code but the bug is still there. This dates 25 years back, to the release of Mac OS X. We start getting a grip on the situation, the bug was there since the first release of Mac OS X shortly after the internet boom. Which means all modern macOS and iOS software were affected by this bug. Nice, so everyone is impacted but I still want to find the origin of the bug.
The XNU kernel released in 2001 is a continuation of the Rhapsody kernel which was based on Mach 2.5 and 4.4BSD kernels. Mach was always the core and the BSD layer built on top of it. Hence, you’d see people refer to XNU as a “hybrid” kernel (2 kernels mashed into 1). But what’s really important is that this kernel is an overhaul of the NeXTSTEP kernel brought into Apple after the acquisition of NeXT in 1997. The NeXTSTEP kernel was first released in 1989 and was based on Mach and 4.3BSD. Since the code was closed-source, it would be much easier to just check the 4.3BSD source code of around that era, 4.3BSD-Tahoe:
// /sys/sys/uipc_usrreq.c
uipc_usrreq(so, req, m, nam, rights)
struct socket *so;
int req;
struct mbuf *m, *nam, *rights;
{
// ...
switch (req) {
// ...
case PRU_SENSE:
((struct stat *) m)->st_blksize = so->so_snd.sb_hiwat;
if (so->so_type == SOCK_STREAM && unp->unp_conn != 0) {
so2 = unp->unp_conn->unp_socket;
((struct stat *) m)->st_blksize += so2->so_rcv.sb_cc;
}
((struct stat *) m)->st_dev = NODEV;
if (unp->unp_ino == 0)
unp->unp_ino = unp_ino++;
((struct stat *) m)->st_ino = unp->unp_ino;
return (0);
// ...
}
}Instead of functions, uipc_sense() was then a small case snippet in a large switch-case handling all userspace requests of UNIX domain sockets. The funny thing is you can see the XNU 123.5 and 4.3BSD Tahoe logic remains pretty much the exact same way, so it’s pretty safe to say this bug survived all the way through the NeXT-Apple era. That’s 12 years back from 2001 to 1989, hot damn!
Digging further into some historic “commits” (because there wasn’t any git at that time), it seems that unp_ino++ line was last changed at December 20, 1985 (unix-history-repo commit 18a9fea), where the previous code was:
case PRU_SENSE:
((struct stat *) m)->st_blksize = so->so_snd.sb_hiwat;
if (so->so_type == SOCK_STREAM && unp->unp_conn != 0) {
so2 = unp->unp_conn->unp_socket;
((struct stat *) m)->st_blksize += so2->so_rcv.sb_cc;
}
((struct stat *) m)->st_dev = NODEV;
((struct stat *) m)->st_ino = unp_ino++;
return (0);It seems like before that commit, inodes of UNIX sockets were fundamentally broken: handing out a different inode each fstat() call. And the change before that in this area was at May 28, 1985 (commit 628f1f5):
case PRU_SENSE:
((struct stat *) m)->st_blksize = so->so_snd.sb_hiwat;
if (so->so_type == SOCK_STREAM && unp->unp_conn != 0) {
so2 = unp->unp_conn->unp_socket;
((struct stat *) m)->st_blksize += so2->so_rcv.sb_cc;
}
return (0);No st_dev or st_ino at all, fstat() on UNIX domain sockets returned plain zeroes. What took my attention was that the commit message of this May 28 change (probably extracted from the version control software of that time) was “fake up inode numbers and dev for the naive”. While we can’t know for sure what the original developer meant by that, I think I was just called naive by a 41-year-old commit message.
The story we can infer here was that at around May 1985 the first userspace program relied on fstat() for UNIX domain sockets on BSD, and someone cared enough to ask for it to be implemented. Furthermore, that someone cared so much that in December 1985, they asked for the inodes to be consistent for the same socket (as you’d expect) and not just blurt out ever-incrementing numbers for each fstat() call. It also seems like that someone was “naive” in the eyes of the kernel developer haha. Well, I can’t know for sure what happened in Berkeley 40 years ago, but it seems like inodes for UNIX domain sockets were really a little feature that may have been prototyped experimentally but wasn’t really re-visited properly. It’s really nice seeing how that little situation moved from the labs of Berkeley over the Bay Bridge to Cupertino and made its mark in the latest versions of the iPhone operating system.
Takeaways
Verify demos better before going on the DEFCON main stage.