backing up your xen domains

backups are boring, but we all know how important they are. backups can also be quite powerful when working with xen virtualization, since xen allows for convenient back-up and restore of entire systems.

i've recently been working on a flexible, general-purpose script enabling incremental backups of complete xen guests, optimized for secure, distributed environments; xenBackup. if you're working with xen, you might find it useful.

the xenBackup script leverages open-source components like ssh, rsync, and rdiff-backup to create a simple, efficient and functional solution.

all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications. if you're unfamiliar with xen, you might consider starting with an earlier how-to on setting up xen on your debian etch box

a general approach

the approach you take to backups obviously depends on what your guests are doing. let's consider one of the more difficult cases, backing up a xen guest with an application server and a database running on it. ideally, you'd typically:
  • take regular backups of the database.
  • take a regular incremental backup of the entire machine.
  • write the backups onto a different server on your network, and hopefully to a different geographical location too.
  • do all of this without any interruption of your service.

in this article we'll discuss a simple way of doing this.

the xenBackup script

the xenBackup script, which i've included at the end of this article, helps implement a xen backup strategy. it automates the backup of single or multiple xen guests using one of three backup methods, tar, rsync or rdiff-backup.

the usage message for xenBackup is as follows:

Usage: xenBackup [OPTION]...
Backup xen domains to a target area. different backup engines may be specified to
produce a tarfile, an exact mirror of the disk area or a mirror with incremental backup.

   -d      backup only the specified DOMAIN
   -t      target LOCATION for the backup e.g. /tmp or root@www.example.com:/tmp
           (not used for tar engine)
   -a      backup all domains
   -s      shutdown domains before backup (and restart them afterwards)
   -q      run in quiet mode
   -e      backup ENGINE to use, either tar, rsync or rdiff-backup
   -p      purge increments older than TIME_SPEC. this option only applies
           to rdiff-backup, e.g. 3W for 3 weeks. see "man rdiff-backup" for
           more information

to illustrate how it could be used, let's consider a typical scenario.

scenario: a single xen server with multiple xen guests

consider a xen server with multiple guests running on it, where the database on each guest backs up locally using a database specific backup technique e.g. a regularly scheduled hot backup writing to the local file system.

xenBackup could be used to periodically backup each of the local guests from the dom0. this is safe to do on a running server since the database backup does not rely on datafile consistency, but instead on the hot backups.

alternatively, the hot backup could be avoided if each of the guests was cleanly shutdown before the backup. xenBackup supports both these modes of operation but the former is recommended.
the xenBackup command on dom0 to incrementally backup all guests to /var/backup would simply be:

$ sudo xenBackup -a -e rdiff-backup -t /var/backup

this arrangement is shown in the diagram above. additionally, this backup could be periodically pulled from another backup server using rsync over ssh. this backup server could be located on or off-site.

this could be simplified by writing the backup directly onto another server in a single xenBackup command. this is arguably less secure since you need to push the backup rather than pull it, but could be done with:

$ sudo xenBackup -a -e rdiff-backup -t root@backupserver:/var/backup

running xenBackup on multiple dom0's the following arrangement can easily be achieved:


the backups

one of the great things about xen is that the backup allows you to reconstitute a fully working xen guest from the backup area, simply with a command like:
$ sudo xen-create-image --copy=/var/backup/mymachine.rdiff-backup.mirror --ip=192.168.1.10 --hostname=mymachine

if rdiff-backup is used as the backup engine, the xen guest can easily be restored to a historical state, to as far back as increments are kept (controlled by the xenBackup purge flag, -p). see man rdiff-backup for more information on using --restore-as-of on your backup directory.

dependencies

the xenBackup script has dependencies on rsync and rdiff-backup, if you choose to use those engines. if you do, you should install those packages:
$ sudo apt-get install rsync rdiff-backup

automation

to automate your backups, consider adding a cron entry to automatically run xenBackup on your backup server. typically you should do this at a quiet time. an example cron entry is:
00 1 * * * /usr/bin/xenBackup -q -a -t /var/backup -e rdiff-backup

you should consider adding the backup server's identity file to the authorized keys of the backup user on machines to push backups to. to allow the automation of encrypted, automated backups to be pushed over ssh. read up on ssh-copy-id and ssh-keygen for more information. give very careful consideration to security when determining the user to use and what machines can access what.

a word on syslog

xenBackup logs all backup output to syslog's local3 facility. all xenBackup's log output is available in the main syslog log, /var/log/syslog. in addition, a dedicated xenBackup log can be created by adding the following to your /etc/syslog.conf file:
# xenBackup logging: log all local3's messages to /var/log/xenBackup
local3.*                        /var/log/xenBackup

if you'd like to be informed about critical backup problems by email, please refer to my earlier blog how to setup real-time email-notification for critical syslog events.

xenBackup - the source

23 aug 2011 -- moved the source to gitHub since i still get a lot of suggestions for changes. have at it. https://github.com/doofdoofsf/xenBackup

Nice script ! However, i've

Nice script !

However, i've got a silly issue :p how can i restore data saved through the dd option ( LVM )

Here my traces from backup :

xenBackup -e dd -d "bdd02.swiatubezpieczen.net" -t /opt/xen-domu-backup

xen-backup: samedi 24 juillet 2010, 00:43:18 (UTC+0200)
xen-backup: backing up domain: bdd02.swiatubezpieczen.net
Logical volume "bdd02.swiatubezpieczen.net-disk-tempsnap" created
xen-backup: backing up domain bdd02.swiatubezpieczen.net to /opt/xen-domu-backup/bdd02.swiatubezpieczen.net.24jui10.4195.img.bz2 using dd and compressing with bzip2
2621440+0 enregistrements lus
2621440+0 enregistrements écrits
5368709120 bytes (5,4 GB) copied, 569,162 s, 9,4 MB/s
xen-backup: SUCCESS: domain bdd02.swiatubezpieczen.net backed up
xen-backup: Cleaning up
Logical volume "bdd02.swiatubezpieczen.net-disk-tempsnap" successfully removed
xen-backup: SUCCESS: backup of all domains completed successfully

But how can i reimport the saved LV to restart my VM ??

Can i use the xm-create-image with special args ? or that make sense ?

used xen 3.2...

Cheersw

Thanks for the script: here

Thanks for the script:
here is a modified version that supports creating snapshots and disk.img-es.

https://blog.valqk.com/archives/Backup-xen-lvmimage-disks.-xenBackup-scr...

Thank you for the

Thank you for the script.
But with clvm (clustered lvm) we can't use snapshot lvm !
Perhaps, it is usefull to maintain a *without snapshot* version of the script ?
In this case, don't you think it is usefull to "xm pause" domU before mounting
and rdiff-backup ?

Cleaned up the code and

Cleaned up the code and fixed some issues with rdiff-backup. This is now the latest version. I've posted it on pastebin:

http://pastebin.com/f3b2a5de6

I found several small issues

I found several small issues (such as, you need to create the snapshot directory on the *remote* host, not the local host when doing a remote backup with rdiff-backup). I also added a few other things to make it work well for an rdiff-backup solution. It's now cleaned up a bit and properly tabbed too. I hope some others find this useful. Enjoy!

I've posted it here: http://pastebin.com/f3b2a5de6

Thanks John for this great,

Thanks John for this great, simple and easy to understand script !

Your work is much

Your work is much appreciated! I'm editing your script to fit my flat disk images setup easily and with very good results.

Regards,
Alessandro

Hi, Thank you for posting

Hi,

Thank you for posting this very useful script, however, I haven't a clue how to adapt the script to use flat files, can anyone point me in the right direction please? Also, could a backup of flat files be taken without stopping the domU's?

Regards,

Alan

thankyou for your hard work

thankyou for your hard work John.
All my domu's are flat files and stored at .img i dont use LVM.

i have a question , does the script only backup LVM based xen disks ?

yes, but it should be a

yes, but it should be a simple modification to use your flat files.

The LVM snapshots addition

The LVM snapshots addition sound just what I am looking for - but is there any documentation on how to use it?

Very impressive script! But

Very impressive script!

But I don't understand why

xenBackup -a -e rdiff-backup -t /var/backup

is a safe back up? Won't the file system be in an unsafe state? I.e. corrupted files.

And what about databases or other programs that cache their data before writing it? That cache that is in the memory won't get backed up...

Lots of love,
Louise

Louise, I address that point

Louise, I address that point in the article:

xenBackup could be used to periodically backup each of the local guests from the dom0. this is safe to do on a running server since the database backup does not rely on datafile consistency, but instead on the hot backups.

alternatively, the hot backup could be avoided if each of the guests was cleanly shutdown before the backup. xenBackup supports both these modes of operation but the former is recommended.

Hi, Very comprehensive

Hi,

Very comprehensive script. One thing I am having trouble with is what do you mean by the "xenDiskArea" var?

My domU images are in /var/lib/xen/images and I've been playing around with the var above but no luck.

Thanks for the script, its

Thanks for the script, its saved me a chunk of time :-)

I've made a few basic modifications to it which some others may find useful - unfortunately its not too pretty and makes an assumption that any VMs living in your volume group have a name ending "-disk" (much like xen-tools creates by default) but much like before feel free to hack it to death to make it do what you want. The functions I've added are:

  • Use of LVM snapshots. (means you can guarantee data consistency for the backup if you're doing it from a running VM)
  • NTFS remounting support (enables me to rsync files out of an NTFS Windows disk image)
  • "all" now backs up all logical volumes in the volume group with a suffix of "-disk" as opposed to just the running domUs. Not pretty but its how we have our system set up.
  • added a "dd" option to backup directly to a compressed image of a block device. Once again this one is aimed at giving me a working copy of a Windows disk image to rsync over to a DR site.

I've just finished tidying a couple of bits up in it so there is a slight chance I've just broken it, but it should be OK. You can download it here.

Nice script and nice

Nice script and nice enhancement. One note though, after taking the LVM snapshot and before taking the backup I would restart the VM if it were shutdown.

thanks for posting this. the

thanks for posting this. the addition of lvm snapshots is a great idea.

Hi! Which tool did you use

Hi!

Which tool did you use to draw the diagrams above ? They really look nice.

Thanks.

visio

visio

Hello, this script is great

Hello,
this script is great but iam experiencing some problems:

backing up domUs which ive created with debootstrap work well, but if i try to backup an domU which was created out of an backup i keep getting errors:

$ xenBackup -e rdiff-backup -d test -t /var/backup
xenBackup: backing up domain: test
xenBackup: Mounting /dev/lvmxen/testt-disk read-only
xenBackup: backing up domain test to /var/backup/test.rdiff-backup.mirror using rdiff-backup
Fatal Error: File /mnt/xen does not look like an increment file.

Perhaps u can give me a hint.

with kind regards

weez

weez, perhaps you could

weez, perhaps you could contact me directly and let me know how you are creating your domU from backup and i'll point you in the right direction.

hi, i think i solved

hi, i think i solved it...

after creating a domU from backup with copy i forgot to remove /rdiff-backup-data/ on my new domU, now its working fine.

weez

Great script, thanks. I just

Great script, thanks.

I just wondering why you don't use LVM snapshots but "mount -r".
Isn't the backup more consistent with snapshots, or is it just to reduce disk space needs ?

Marc

marc, to be honest, i didn't

marc, to be honest, i didn't consider it. it sounds like a good alternative, albeit with the possible downside that you point out.

any interested in posting a patch to add this as an option?

Great script, I was looking

Great script, I was looking for a good way to backup my xen domains, thanks again! John!

Looks great, though it could

Looks great, though it could use support for loop mounted disk images :)

Awesome script work, John!

Awesome script work, John! Thanks!

Please note, this entry has been closed to new comments.