Advanced Namespace Tools blog

11 January 2018

Plan 9 has had independent ip addresses all along

Years ago I wrote “almost everything I want to do with computers can be done with a few lines of rc in Plan 9” and I just re-learned that lesson very powerfully, and got a good lesson in the difference between vague, superficial understanding and real comprehension. For a long time, I had thought that the idea of giving separate ip addresses to the independent namespaces ANTS creates was some kind of difficult-to-implement feature that I’d have to tackle at some point in the future. Nope, Plan 9 has always had this available, I was just too dense to get what the manpages were telling me. Thanks to the 9front irc crew for clearing up my confusion.

How VMX handles networking

After always being frustrated by the complexity of virtual machine networking in Linux, with more or less incomprehensible tun/tap command strings and vm configurations that I was rarely able to get to work right even following along with guides, the ease and simplicity of networking vmx virtual machines in 9front was refreshing - with no configuration needed other than telling the vm to use the host ethernet, the vm would happily claim its own IP address via DHCP and everything just worked naturally.

This led me to thinking “well, since this works so nicely for a vm, maybe we could have the same functionality in userpsace?” I started poking around in the vmx code, trying to understand how this magical trick of letting the vm get an independent IP on the same box using the same ethernet address worked. Being a Bear of Little Brain I didn’t get very far with this, mostly ending up very impressed with the fact that Aiju had created the vmx implementation as a solo effort in a short time.

I expose my foolishness in IRC

So, thinking I might have thought up something clever and useful, I ask in irc if it would be possible to extract this magical code from vmx to allow userspace processes to communicate on independent IP addresses without a full virtualization layer. The response from those in-the-know was unanimous. “WTF are you talking about? vmx doesn’t do what you think it does.” After a little bit of confusion where I asked questions based on my misunderstandings, the real picture was patiently explained to me. I’m clearly not very good at getting at the essence of things from reading manpages, because once I started to get it via the dialectic of irc, a succession of “aha! so THATS what all that stuff in the manpages I’ve read so many times” moments followed.

Namespacing multiple addresses on the same IP stack

The way to achieve the goal is just to bind a new ip address onto an existing IP configuration, and then direct listeners to use it. So, my box is DHCP to a given address in the 192.168.0.x range by my LAN router. I can just find an unused ip address and do:

   ip/ipconfig -g 192.168.0.1 ether /net/ether0 add 192.168.0.99 255.255.255.0
   aux/listen1 tcp!192.168.0.99!2500 /bin/fortune

And then from another box, ifI do

   telnet tcp!192.168.0.99!2500

I receive a random fortune. All that is happening here is that we are adding a new name, 192.168.0.99, to an existing ip stack on an interface without removing the previous address, and then telling the listener to only listen for connections to that ip-name. The usual central parameter in aux/listen is usually ‘*’ which directs the listener to use all interfaces, but by giving it a specific number, we can have multiple listeners on the same port on the same box, but using different ip address namespaces.

Digging into the kernel devices

This is a good opportunity to understand the underlying mechanisms a little more clearly. Basic network config in Plan 9 works like this:

   bind -b '#l0' /net
   bind -b '#I0' /net
   ip/ipconfig          #standard DHCP, or
   ip/ipconfig -g ip.gateway ether /net/ether0 ip.address net.mask

Let’s pull this apart to see the details of what is going on. The first bind command of #l0 is for the hardware device. /net starts empty, and after the first bind command, contains nothing but the /net/ether0 directory. The next bind command doesn’t refer to the hardware, it refers to the in-kernel ip software stack. It contains a series of protocol directories such as tcp, udp, the il protocol (Plan 9’s original preferred ip protocol) and control and information files like ipifc and iproute and ipselftab. The ipconfig command “glues” the ip software stack together with the hardware interface in a specific configuration.

With multiple hardware interfaces available, it is easy to create multiple ip stacks bound in different places. On my vultr vms, I use this to make the intra-grid service connections on a secondary, private networking interface. At bootup, venti, fossil, and tcp boot cpus configure the secondary interface on /net.alt to use the private subnet. See the blog post on using net.alt.