So apparently I'm behind the curve here. I've been using my own ghetto home brew scripts when there are two beautiful OpenSource alternatives just sitting out there waiting to be used! Check out rdiff-backup and duplicity.
http://fsvs.tigris.org/ (aims to become a complete backup/restore tool for all files in a directory tree or whole filesystems, with a subversion repository as the backend)
The difference is that rdiff-backup aim's to have the incremental's as easy to read as possible, while duplicity keeps all the backups encrypted using GnuPG. So they are perfectly anologous to my two scripts, only better!
See discussion of other useful methods at diveintomark.org and mikerubel.org.
I've been looking for a backup solution which fits my needs for a while (see BackupSoftware). I keep almost all of my personal data on my colocated server which I don't have regular access to. This means that cd/tape methods don't work well since they require changing media on a regular basis and to get backups off site.
The best solution I was able to come up with was a simple Rsync script (which I run every night) which syncs my file system to the filesystem of a friends box in New York. This works great, is bandwidth friendly and convenient (need a backup, just login to the other box and grab the files you want)..
#!/bin/sh -x (-)
RSYNC="rsync -avuz --delete --quiet --bwlimit=128"
export RSYNC_RSH="ssh -q -x -c blowfish"
USER="larry"
HOST="maus.spack.org"
DEST="/usr/local/backups/maus.spack.org"
DIRLIST="/home/larry"
for directory in $DIRLIST; do
${RSYNC} ${USER}@${HOST}:${directory} ${DEST}
done
The big remaining problem was how to sync my personal data, email/writing/etc which I don't want others poking around in without my express permission. Currently I've just been running the same script from my laptop, but since my laptop isn't always online it means that backups only happen as I remember to start them, and I want something which is automated.
Today, thanks to a conversation with my workmate John, I discovered this simple hack. I introduce tar to ssh to gpg. Whee.
#!/bin/sh (-)
# Written by Adam on 27 March 2003
# Make sure all needed binaries are available in the path.
PATH=/bin:/sbin:/usr/bin:/usr/sbin
GPG="gpg --encrypt --default-recipient-self"
TAR="tar zcf -"
HOST="larry@maus.spack.org"
DIRS="/home/ashand/tmp /tmp/user_files"
OUTPUT="/tmp/foo.tar.gz.gnupg"
${TAR} ${DIRS} | ssh ${HOST} "${GPG} -o ${OUTPUT}"
This leaves you with an encrypted, compressed tarball on the remote server. Despite rumors to the contrary accessing the data within is quite easy. By default GnuPg sends output to standard out so you just decrypt it and pipe it to tar to process as you wish.
maus# gpg -d /tmp/foo.tar.gz.gnupg | tar ztf -
You need a passphrase to unlock the secret key for
user: "Adam Shand <larry@...>"
2048-bit ELG-E key, ID 161B05D8, created 2000-10-28 (main key ID 8F07D4C2)
gpg: encrypted with 2048-bit ELG-E key, ID 161B05D8, created 2000-10-28
"Adam Shand <larry@...>"
home/ashand/tmp/
home/ashand/tmp/scratch
... <snip> ...
Notes:
- Thanks to Brendan Murray from Otago University for the tar pipe tar magic all those years ago.
Both scripts assume that you have passwordless ssh (see UsingSsh) setup in order for it to be run from cron.