Joe’s Amazing Technicolor Weblog

I’m putting together our backup testing plan, and marveling at the suggestions in Preston’s Backup and Recovery. Here’s my paraphrase:

• restore many single files
• restore older versions of files
• restore entire drive / filesystem, compare to original (same size? etc.)
• recreate entire system
• pretend a given backup volume is bad, use alternate
• restore without touching backup server (as if it were destroyed)
• include database restores, inc. database at different point in time
• dream up painful scenarios with pessimists, test for those regularly

To actually do these tests, he suggests making a list & randomly picking a subset to test on a monthly basis.

Fun, huh? Beats holding the bag when your organization’s vital data goes missing.

Because rsync 3 is one of if not the only OS X backup solutions that actually gets all the possible forms of metadata, I’m in the market for a backup solution that uses it. The current candidate is rsnapshot, a 6000 line perl program (!), conveniently located in macports, that wraps rsync to do smart backup things like keep snapshots via hard links. Thanks to the O’Reilly Backup book for pointing me to it.

• It’s pretty easy to set up to run locally. The main trick is that the configuration file requires tabs. First time in ~10 years I’ve had to turn tabs back on in vim (:set noexpandtab).
• For OS X, we want the magic -aNHAXx --fileflags --force-change args to make rsync behave properly with all the metadata.
• OS X has a weird directory structure, so if you try to backup /etc you just get the symlink that is to /private/etc, or if you try to exclude something under /var, you miss it because it’s really /private/var/bigdirectory. Buzzkill.
• Lchown.pm is necessary for symlinks to have the right ownership in snapshots. CPAN’d.
• Running things automatically on OS X as the privileged user is a bit odd, lacking as it does a traditional root account. Sudo does nicely, with NOPASSWD: /opt/local/bin/rsnapshot
on the backup machine. For the clients we need appropriate ssh settings, with some tricks to run sudo on the remote machine. Getting this running took a while, since I missed the fact that running rsnapshot via sudo on the backup machine meant that rsync would try to use root’s ssh key, not the backup user’s — fixed this with the -i arg to ssh.
• Restoring backups is just a matter of copying them from the appropriate snapshot dir, probably using rsync & the same arguments rsnapshot uses (easily extracted from the rsnapshot log).
• Multiple servers are done serially. If you wanted to run backups in parallel, you’d need one configuration file for every server you’re backing up, and they each need their own snapshot_root, logfile, and lockfile. For my installation, this’d be more trouble than it’s worth.
  

Another issue that temporarily gave me pause, now that we’ve got everything backed up with presumably correct permissions, is that our off-site backup procedures involve creating tar files & encrypting them. Fortunately, backup bouncer shows that OS X tar gets all the important stuff right, though it’s no rsync v3.

Update: I’d forgotten to check the flags that had been missing from the macport version last time around, but they’re all there. Now, running rsync -aNHAXx --fileflags --force-change /Volumes/Src/ /Volumes/rsync3test passes every bbouncer test! Cool.

Original post: On Leopard (10.5.5) using rsync 3.0.4 from macports, a few more tests are passing backup bouncer 0.1.3 (compare to my June post):

$ sudo ./bbouncer verify -d /Volumes/Src/ /Volumes/rsync3test/
Verifying:    basic-permissions ... ok (Critical)
Verifying:           timestamps ... ok (Critical)
Verifying:             symlinks ... ok (Critical)
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok (Important)
Verifying:       resource-forks ...
   Sub-test:             on files ... ok (Critical)
   Sub-test:  on hardlinked files ... ok (Important)
Verifying:         finder-flags ... ok (Critical)
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok (Important)
   Sub-test:       on directories ... ok (Important)
   Sub-test:          on symlinks ... ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok (Important)
   Sub-test:              on dirs ... ok (Important)
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok

As regular readers saw earlier, I’ve been casting about trying to find an open source backup solution that handles OS X metadata reliably. Having been disappointed by rdiff-backup, I’ve turned by eyes to the similar rsnapshot project, which uses the venerable rsync.

Since macports includes the latest version of rsync, 3.0.2, I gave it a try with the familiar rsync -avz /from /to syntax, but it performed disappointingly on n8’s handy Backup Bouncer test suite. Thanks to Mike Bombich, I learned about some extra flags to add (though my copy of rsync doesn’t seem to know about the -N or — fileflags he has):

$ sudo rsync -aHAXx  /Volumes/Src/ /Volumes/rsynctest/
$ ./bbouncer verify -d /Volumes/Src/ /Volumes/rsynctest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok
Verifying:                 fifo ... FAIL
Verifying:              devices ... FAIL
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... ok
ok

Sure, there are a few FAILS in there, but they’re not important:

$ ./bbouncer verify -T important -d /Volumes/Src/ /Volumes/rsynctest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... ok
ok
Verifying: access-control-lists ...
   Sub-test:             on files ... ok
   Sub-test:              on dirs ... ok
ok

(note the -T important flag telling Backup Bouncer to remove the extra-finicky tests). Good enough! On to get familiar with rsnapshot.

I’ve finally had the chance to repeat my test of the rdiff-backup-devel package from MacPorts, using the same steps as I used for the stable package. The results are better but not fantastic:

$ ./bbouncer  verify -d /Volumes/Src/ /Volumes/rdifftest
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... FAIL
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... ok
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... ok
   Sub-test:       on directories ... ok
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
FAIL
Verifying:                 fifo ... ok
Verifying:              devices ... ok
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... ok
   Sub-test:     lots of metadata ... FAIL
FAIL

With promising reports out on rsync 3, looks like it’s time to take another look at rsnapshot…

Preamble: after a bunch of research, rdiff-backup looked like a good open source solution for OS X backup. And it’s in macports!

Wrinkle: after getting further into my current backup project, I’ve become more aware of the difficulty of getting all of the various forms of OS X metadata backed up. Some research revealed the tool Backup Bouncer, which provides a way to run some tricky tests on any OS X backup system. Here I’m using the latest bbouncer (0.1.2), on a 10.4.11 system all patched up through security update 2008-03.

The stable version of rdiff-backup in macports is 1.0.5, which is what’s being used for the following tests. I’ve poked at both this and the rdiff-backup-devel version, which is 1.1.15, but it doesn’t automatically pull in the xattr module from macports. I’ll retest with that version later. Also note that the bbouncer results don’t suggest that xattr is doing 1.0.5 much good.

$ ./bbouncer create-vol rdifftest
$ sudo rdiff-backup /Volumes/Src /tmp/bb
$ sudo rdiff-backup --force -r 1D /tmp/bb /Volumes/rdifftest/
$ ./bbouncer verify -d /Volumes/Src/ /Volumes/rdifftest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:    symlink-ownership ... ok
Verifying:            hardlinks ... ok
Verifying:       resource-forks ... FAIL
Verifying:         finder-flags ... FAIL
Verifying:         finder-locks ... FAIL
Verifying:        creation-date ... FAIL
Verifying:            bsd-flags ... FAIL
Verifying:       extended-attrs ...
   Sub-test:             on files ... FAIL
   Sub-test:       on directories ... FAIL
   Sub-test:          on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
   Sub-test:             on files ... FAIL
   Sub-test:              on dirs ... FAIL
FAIL
Verifying:                 fifo ... FAIL
Verifying:              devices ... FAIL
Verifying:          combo-tests ...
   Sub-test:  xattrs + rsrc forks ... FAIL
   Sub-test:     lots of metadata ... FAIL
FAIL

That’s a whole lot of FAIL, especially compared to the built-in OS X rsync, which passes on resource forks and finder flags, extended attributes for files and directories, ACLs, and the last few tests.

Granted, there are a bunch of things being tested by bbouncer that may or may not be of interest to the average user. A recent post by the author points out that bbouncer has a -T flag for indicating which level of paranoia to run at. Rerunning the test for rdiff-backup 1.0.5 with only critical tests is somewhat instructive:

$ ./bbouncer verify -d -T critical /Volumes/Src/ /Volumes/rdifftest/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:       resource-forks ... FAIL
Verifying:         finder-flags ... FAIL

Compare to OS X’s rsync:

$ ./bbouncer verify -d -T critical /Volumes/Src/ /Volumes/Dst/10-rsync-apple/
Verifying:    basic-permissions ... ok
Verifying:           timestamps ...
   Sub-test:    modification time ... ok
ok
Verifying:             symlinks ... ok
Verifying:       resource-forks ... ok
Verifying:         finder-flags ... ok