Advanced Namespace Tools blog

07 February 2017

Adding End-of-File Messages to Hubfs

A long-standing annoyance with hubfs was the inability to send an eof to programs reading from the hub files. A core aspect of the design is that once programs reading from the hubfile have consumed the buffer, they block on the read syscall waiting for more data to be sent, receive the new data, and block again waiting for more.

This is correct and desired in most situations, but there are a few times it becomes problematic. One is that if you want to use a standard shell tool to process the data in a hub buffer, you need to set the whole hubfs to ‘freeze’ mode, then ‘melt’ it afterwards. A more common frustration results from running a program that consumes standard input - with no way for send an eof through the hubs, the program will never terminate. For instance, it is common to do something like this to write short files:

   cat >somefile

You type what you want in somefile, then press ctrl-d to send eof and end the cat process. In previous versions of hubfs, pressing ctrl-d would just break the input stream of the hubshell and end it, leaving the hub-attached cat running. When you reconnected to the hubfs, the cat would still be there consuming data and redirecting to somefile.

Solving the Problem, Layer 1: Hubfs

The first step in solving this is seeing if we can get an eof transmitted to direct hubfs clients. For reading from an fd with read (2), we can test for end of file as follows:

   n = read(fd, buf, buflen);
   if(n == 0)
          fprint(2, "end of file reached\n");

Since hubfs is a 9p fs serving synthetic files, we can send eof by setting r->ofcall.count to zero when responding to a client Tread. For a first test, I just implemented a global eof to all readers when the string “eof” was written to the ctl file. This sort-of worked, with the problem that until a new write was made to the hub, waiting read requests wouldn’t be serviced. This meant that you had to write “eof” to the ctl file, then send some new data to the hub, and then readers would receive the eof. This was pretty clearly not good enough. The basic logic, inside the msgsend function:

   if(endoffile == UP){
          r->ofcall.count = 0;
          h->rstatus[i] = DONE;
          if((i == h->qrans) && (i < h->qrnum))
                 h->qrans++;
          respond(r, nil);
          continue;
   }

Improving this required adding some new infrastructure to hubfs. Hubfs relies on the 9pfile (2) library for managing basic filesystem services like adding and listing files, which means that previously, it didn’t even really know about what hubfiles were created within it, it just knew how to respond to the reads and writes sent to them. Getting the send-eof-to-all-clients logic right meant that we needed to walk through all active hubs sending eof on each of them, so we need to make a linked list of currently active hubs.

   struct Hublist{
          Hub* targethub;             /* Hub referenced by this entry */
          char *hubname;              /* Name of referenced Hub */
          Hublist* nexthub;    /* Pointer to next Hublist entry in list */
   };

   Hublist* firsthublist;      /* Pointer to start of linked list of hubs */
   Hublist* lasthublist;       /* Pointer to the list entry for next hub to be created */

Hubfs has no default hubs created when it starts, so we begin by making an empty Hublist entry in main, and our function to add a hub to the list will always create a new empty Hublist entry that lasthublist points at. In main() fs initialization section:

   /* start with an allocated but empty first Hublist entry */
   lasthublist = (Hublist*)emalloc9p(sizeof(Hublist));
   lasthublist->targethub = nil;
   lasthublist->hubname = nil;
   lasthublist->nexthub = nil;
   firsthublist = lasthublist;

In addhub(Hub *h), called after zerohub during fscreate():

   lasthublist->targethub = h;
   lasthublist->hubname = h->name;
   lasthublist->nexthub = (Hublist*)emalloc9p(sizeof(Hublist)); /* always keep an empty */
   lasthublist = lasthublist->nexthub;
   lasthublist->nexthub = nil;
   lasthublist->targethub = nil;

Now we can write eofall() to walk through the hubs calling msgsend(Hub *h) on each hub:

   Hublist* currenthub;
   currenthub = firsthublist;
   if(currenthub->targethub == nil)
          return;
   msgsend(currenthub->targethub);
   while(currenthub->nexthub->targethub != nil){
          currenthub=currenthub->nexthub;
          msgsend(currenthub->targethub);
   }

Great, this works! But we really don’t want to send eof to ALL connected hubs very often, what we want to do is send it to a particular hub of our choice. So, the ctl file should accept a message like “eof hubname” and then dispatch to only that selected hub. Here’s the ctl file logic:

   /* eof command received, check if it applies to single hub then call matching eof func */
   if(strncmp(cmd, "eof", 3) == 0){
          endoffile = UP;
          if(strlen(cmd) > 4){
                 i=0;
                 while(isalnum(*(cmd+i+4))){
                        cmdbuf[i]=*(cmd+i+4);
                        i++;
                 }
                 cmdbuf[i] = '\0';
                 eofhub(cmdbuf);
                 endoffile = DOWN;
                 return;
          }
          fprint(2, "hubfs: sending end of file to all client readers\n");
          eofall();
          endoffile = DOWN;
          return;
   }

And the eofhub(char *target) logic:

   Hublist* currenthub;

   currenthub = firsthublist;
   if(currenthub->targethub == nil)
          return;
   if(strcmp(target, currenthub->hubname) == 0){
          fprint(2, "hubfs: eof to %s\n", currenthub->hubname);
          msgsend(currenthub->targethub);
          return;
   }
   while(currenthub->nexthub->targethub != nil){
          currenthub=currenthub->nexthub;
          if(strcmp(target, currenthub->hubname) == 0){
                 fprint(2, "hubfs: eof to %s\n", currenthub->hubname);
                 msgsend(currenthub->targethub);
                 return;
          }
   }

There’s one more thing that we need. Hubs can be removed, so we need to deal with removing Hublist entries when fsdestroyfile is called. It calls removehub(Hub *h)

   Hublist* currenthub;

   currenthub = firsthublist;
   if(currenthub->targethub = h){
          if(currenthub->nexthub != nil)
                 firsthublist = currenthub->nexthub;
          free(currenthub);
          return;
   }
   while(currenthub->nexthub->targethub != nil){
          if(currenthub->nexthub->targethub = h){
                 currenthub->nexthub = currenthub->nexthub->nexthub;
                 free(currenthub->nexthub);
                 return;
          }
          currenthub=currenthub->nexthub;
   }

The fact that there is always an “empty” final Hublist entry causes some conceptual confusion in the linked list handling. In fact, looking at the above code, I think the second if() statement can never fail, because if there is an active Hub, there will always be an available next Hublist (possibly the empty one).

So, we now have the ability to eof a specific hub via a ctl message, but this isn’t maximally convenient as a user. What we would like is to be able to have an identical interface when working in a shell via hubshell - press ctrl-d to eof the active subprogram.

Solving the Problem, Layer 2: Hubshell

Hubshell acts as management logic for using hubfs conveniently. It reads and writes the same hubfiles that an rc is attached to, bucket-brigades the data to and from the user’s standard io, and can create and move between new hubfs-connected rcs. The previous logic for the process reading and writing fd0 would break the read/write loop when a zero-length read was detected and exit. The new logic is:

   readloop:
   while((n=read(infd, buf, (long)sizeof(buf)))>0){
          [omitted logic for %command dispatch]
          sleep(s->fdzerodelay);
          if(write(outfd, buf, n)!=n)
                 fprint(2, "hubshell: write error copying on fd %d\n", outfd);
          if(s->shellctl == 'q')
                 exits(nil);
   }
   /* eof input from user, send message to hubfs ctl file */
   if(n == 0){
          if((ctlfd = open(ctlname, OWRITE)) == - 1){
                 fprint(2, "hubshell: can't open ctl file\n");
                 goto readloop;
          }
          sprint(ctlbuf, "eof %s\n", basehub);
          n=write(ctlfd, ctlbuf, strlen(ctlbuf) +1);
                 if(n != strlen(ctlbuf) + 1)
                        fprint(2, "hubshell: error writing to %s on fd %d\n", ctlname, ctlfd);
          close(ctlfd);
          goto readloop;              /* Use more gotos, they aren't harmful */
   }

Getting all this to work required some new logic to get the “basehub” and “ctlname” strings set correctly from the name of the hub, which is all done with kind of annoying c string handling logic like this:

   sprint(basehub, s->fdname[0] + strlen(hubdir));

Where hubdir and ctlname are set up like this:

   /* parse initname and set srvname hubdir and ctlname from it */
   strncpy(initname, argv[1], SMBUF);
   strncat(srvname, initname+3, SMBUF);
   sprint(srvname + strcspn(srvname, "/"), "\0");
   sprint(hubdir, "/n/");
   strncat(hubdir, srvname, SMBUF-6);
   strcat(hubdir, "/");
   sprint(ctlname, "/n/");
   strncat(ctlname, srvname, SMBUF-6);
   strcat(ctlname, "/ctl");

Since I had added logic to hubshell that involved writing to the hubfs ctl file, I figured I should add the ability to pass more ctl messages through the hubshell %commands, so you can now set things like the paranoid/calm mode flag via %fear and %calm.

Doing all this took way longer than it should have, because of my talent for inserting typos/logic errors into standard boilerplate code. I probably spent two or three hours debugging the following typo:

   if(ctlfd = open(ctlname, OWRITE) == - 1){

The missing parenthesis (compare with correct version further up) result in ctlfd being set to 0, which caused a lot of fun and misleading problems. I knew perfectly well what the correct form was, and had written in dozens (hundreds?) of times previously, so my eye just skipped right over that line as “obviously standard and correct”.

Anyway, fun bugs aside, getting eof messages passed through hubshell to hubfs definitely improves usability. Thanks to sam-d for reminding me that this was something I should get working!