backing up your xen domains

backups are boring, but we all know how important they are. backups can also be quite powerful when working with xen virtualization, since xen allows for convenient back-up and restore of entire systems.

i've recently been working on a flexible, general-purpose script enabling incremental backups of complete xen guests, optimized for secure, distributed environments; xenBackup. if you're working with xen, you might find it useful.

the xenBackup script leverages open-source components like ssh, rsync, and rdiff-backup to create a simple, efficient and functional solution.

all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications. if you're unfamiliar with xen, you might consider starting with an earlier how-to on setting up xen on your debian etch box

a general approach

the approach you take to backups obviously depends on what your guests are doing. let's consider one of the more difficult cases, backing up a xen guest with an application server and a database running on it. ideally, you'd typically:

• take regular backups of the database.
• take a regular incremental backup of the entire machine.
• write the backups onto a different server on your network, and hopefully to a different geographical location too.
• do all of this without any interruption of your service.

in this article we'll discuss a simple way of doing this.

the xenBackup script

the xenBackup script, which i've included at the end of this article, helps implement a xen backup strategy. it automates the backup of single or multiple xen guests using one of three backup methods, tar, rsync or rdiff-backup.

the usage message for xenBackup is as follows:

to illustrate how it could be used, let's consider a typical scenario.

scenario: a single xen server with multiple xen guests

consider a xen server with multiple guests running on it, where the database on each guest backs up locally using a database specific backup technique e.g. a regularly scheduled hot backup writing to the local file system.

xenBackup could be used to periodically backup each of the local guests from the dom0. this is safe to do on a running server since the database backup does not rely on datafile consistency, but instead on the hot backups.

alternatively, the hot backup could be avoided if each of the guests was cleanly shutdown before the backup. xenBackup supports both these modes of operation but the former is recommended.
the xenBackup command on dom0 to incrementally backup all guests to /var/backup would simply be:

this arrangement is shown in the diagram above. additionally, this backup could be periodically pulled from another backup server using rsync over ssh. this backup server could be located on or off-site.

this could be simplified by writing the backup directly onto another server in a single xenBackup command. this is arguably less secure since you need to push the backup rather than pull it, but could be done with:

running xenBackup on multiple dom0's the following arrangement can easily be achieved:

the backups

one of the great things about xen is that the backup allows you to reconstitute a fully working xen guest from the backup area, simply with a command like: if rdiff-backup is used as the backup engine, the xen guest can easily be restored to a historical state, to as far back as increments are kept (controlled by the xenBackup purge flag, -p). see man rdiff-backup for more information on using --restore-as-of on your backup directory.

dependencies

the xenBackup script has dependencies on rsync and rdiff-backup, if you choose to use those engines. if you do, you should install those packages:

automation

to automate your backups, consider adding a cron entry to automatically run xenBackup on your backup server. typically you should do this at a quiet time. an example cron entry is: you should consider adding the backup server's identity file to the authorized keys of the backup user on machines to push backups to. to allow the automation of encrypted, automated backups to be pushed over ssh. read up on ssh-copy-id and ssh-keygen for more information. give very careful consideration to security when determining the user to use and what machines can access what.

a word on syslog

xenBackup logs all backup output to syslog's local3 facility. all xenBackup's log output is available in the main syslog log, /var/log/syslog. in addition, a dedicated xenBackup log can be created by adding the following to your /etc/syslog.conf file: if you'd like to be informed about critical backup problems by email, please refer to my earlier blog how to setup real-time email-notification for critical syslog events.

tech blog

if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.

xenBackup - the source

here's the source for the backup script. please feel free to use, modify, plagiarize, mock, torture or hack up any of this bash code as your mood takes you.