Advanced Namespace Tools blog

22 January 2017

Mutual Backup of Systems with Venti

I recently added a new native terminal system to my home grid, which presented a good opportunity to improve my backup workflow for my local systems. In general, I keep “important” things backed up between different remote VPS nodes using more or less the same system described here. My local system is mostly for scratch work/experiments/fun and so I was handling backup via occasional dumps of the venti or ad-hoc copying of pieces of the filesystem somewhere. With another local native system, though, it makes sense to move to a more thorough and robust backup flow.

Filesystems vs. Deduplicating Data Buckets

With hypothetical “ordinary naive” filesystems, you will have a bad time if you try to make a daily cronjob which does something like this for a backup system:

   cp -r /fsAlpha /fsBeta/backupfs1
   cp -r /fsBeta /fsAlpha/backupfs2

This is going to crash and burn after a few days, because it causes an exponential explosion of duplicated data: since fsBeta contains a full copy of fsAslpha, fsAlpha ends up with two copies of its original data after the first day, then four after day two, eight after day three, and similarly for fsBeta as well. (Yes, I know that nobody is dumb enough to do this, and you can just make dedicated partitions on the different machines to avoid the potential recursive duplication problem.)

Venti in Plan 9 is not a filesystem on its own; its a deduplicating block data store, and an analogous process to the above actually works fine, and is a nice way to have two (or potentially more) systems keep an updated backup of the other systems available. The next section shows how it works in practice.

Setting up Progressive Venti Mirroring

The goal is to keep multiple ventis synchronized with up-to-date data, so that all of their clients (possibly active fossils, possibly vacfs backups of any other filesystem) can make use of any of the available ventis in the event that their primary venti becomes unavailable. The method for doing so is a small script called “ventiprog” which makes use of the venti/wrarena command described in the venti-backup Plan 9 manpage.

The wrarena command tends to look a bit scary. Here is a sample:

   venti/wrarena -h tcp!192.168.0.99!17034 -o 1611407360 /dev/sdE0/arenas 0x442b1b4

It’s because this commnand is a bit gnarly that it is nice to use a wrapper script for manipulating it. The meaning of the parameters are, in order: the address of the venti server we are copying TO, the offset within the arenas partition of the particular arena we are copying from, the location of the arenas partition, and the starting location of the data blocks we want to copy within that partition.

The ANTS ventiprog system conventionally stores the wrarena command as ‘wrcmd’ within the 9fat partition. When ventiprog is run, it updates the wrcmd with a new final parameter. Handling the arena parition -o parameter is currently done manually when an arena is filled, but I should really automate that part as well. Under the current system, here is how backing up a Venti and keeping the copy current works:

First, run the script ‘ventioffsets’ located in the scripts subdirectory of ANTS. It is an excerpt from the example venti backup script in venti/words/backup.example in the standard plan 9 distribution. It produces output like this:

   cpu% ventioffsets
   x arenas0 794624 /dev/sdE0/arenas
   x arenas1 537665536 /dev/sdE0/arenas
   x arenas2 1074536448 /dev/sdE0/arenas
   x arenas3 1611407360 /dev/sdE0/arenas

Those are numbers for the -o parameter for the wrarena command for the arenas within the arenas partition. Supposing that you have your backup venti running with a public listener on your LAN at the address tcp!192.168.0.99!17034, you will start this way:

   echo 'venti/wrarena -h tcp!192.168.0.99!17034 -o 794624 /dev/sdE0/arenas 0x0' > /n/9fat/wrcmd
   ventiprog

The first ventiprog command will send all the data from the first arenas0 partition to the target venti. Then you will need to create the next wrcmd to copy the arenas1 partition using its offset data:

   echo 'venti/wrarena -h tcp!192.168.0.99!17034 -o 53766536 /dev/sdE0/arenas 0x0' > /n/9fat/wrcmd
   ventiprog

Once you have worked through the arenas to be copied, you will end up with a saved wrcmd with a final parameter like 0x442b1b4 as shown above. Now that you have synchronized the backup venti to the current data, you should add a copy of the wrcmd to the 9fat partition of that server also, but with the -h parameter changed to target the original venti. From this point on, you can use the ventis independently, but once per day issue the ‘ventiprog’ command on each server, to copy the new blocks to the other server and update the local wrcmd. Because Venti is deduplicative, only the new and different blocks written since the last ventiprog command will be copied, allowing each Venti to act as an up-to-date backup for the other. When an arena is filled, manually update the wrcmd with the -o parameter of the new arena, and change the final parameter back to 0x0.

Coordinating and Verifying Venti-backed Fossil backup

Fossil has a mixed reputation among Plan 9 users, but I find that in combination with Venti and with appropriate configuration and management, it works well for my systems. I believe the key is treating fossil as a temporary and perhaps unreliable buffer for Venti storage. This means using frequent archival snapshots and keeping track of the rootscores. In the event of a problem, you can reset the fossil to the most recent rootscore with fossil/flfmt -v $rootscore. (That isn’t a literal command unless you have created a $rootscore variable on your own.)

ANTS has a conventional location for storing rootscores: /n/9fat/rootscor. (The final “e” is missing, in the tradition of “creat”.) The “fossilize” command appends the most recent rootscore (acquired from fossil/last) to that file. Assuming fossil is configured to make daily archival snapshots, and an up-to-date wrcmd is saved in 9fat, here is a workflow that updates the backup venti and verifies that the most recent saved rootscore is usable:

   fossilize /dev/sdE0/fossil
   echo vac:blahblahblah >/tmp/verify.vac #blah being the rootscore printed
   ventiprog
   vacfs -h backup.venti.address /tmp/verify.vac

If everything has gone properly, vacfs will run without errors and mount a read-only copy of the backed-up fossil at /n/vac. (Note that the current main tree will be at /n/vac/active). If you get an error like “vacfs: vacfsopen: no block with score blahblahblahblah/16 exists” the most likely cause is that not all the venti data had been written to disk at the time that ventiprog was run. Try waiting a few minutes and re-running ventiprog and see if the final parameter changes, then run vacfs again.