backing up your xen domains

posted february 10th, 2008 by john
in
backups are boring, but we all know how important they are. backups can also be quite powerful when working with xen virtualization, since xen allows for convenient back-up and restore of entire systems.

i've recently been working on a flexible, general-purpose script enabling incremental backups of complete xen guests, optimized for secure, distributed environments; xenBackup. if you're working with xen, you might find it useful.

the xenBackup script leverages open-source components like ssh, rsync, and rdiff-backup to create a simple, efficient and functional solution.

all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications. if you're unfamiliar with xen, you might consider starting with an earlier how-to on setting up xen on your debian etch box

a general approach

the approach you take to backups obviously depends on what your guests are doing. let's consider one of the more difficult cases, backing up a xen guest with an application server and a database running on it. ideally, you'd typically:
  • take regular backups of the database.
  • take a regular incremental backup of the entire machine.
  • write the backups onto a different server on your network, and hopefully to a different geographical location too.
  • do all of this without any interruption of your service.

in this article we'll discuss a simple way of doing this.

the xenBackup script

the xenBackup script, which i've included at the end of this article, helps implement a xen backup strategy. it automates the backup of single or multiple xen guests using one of three backup methods, tar, rsync or rdiff-backup.

the usage message for xenBackup is as follows:

Usage: xenBackup [OPTION]...
Backup xen domains to a target area. different backup engines may be specified to
produce a tarfile, an exact mirror of the disk area or a mirror with incremental backup.

   -d      backup only the specified DOMAIN
   -t      target LOCATION for the backup e.g. /tmp or root@www.example.com:/tmp
           (not used for tar engine)
   -a      backup all domains
   -s      shutdown domains before backup (and restart them afterwards)
   -q      run in quiet mode
   -e      backup ENGINE to use, either tar, rsync or rdiff-backup
   -p      purge increments older than TIME_SPEC. this option only applies
           to rdiff-backup, e.g. 3W for 3 weeks. see "man rdiff-backup" for
           more information

to illustrate how it could be used, let's consider a typical scenario.

scenario: a single xen server with multiple xen guests

consider a xen server with multiple guests running on it, where the database on each guest backs up locally using a database specific backup technique e.g. a regularly scheduled hot backup writing to the local file system.

xenBackup could be used to periodically backup each of the local guests from the dom0. this is safe to do on a running server since the database backup does not rely on datafile consistency, but instead on the hot backups.

alternatively, the hot backup could be avoided if each of the guests was cleanly shutdown before the backup. xenBackup supports both these modes of operation but the former is recommended.
the xenBackup command on dom0 to incrementally backup all guests to /var/backup would simply be:

$ sudo xenBackup -a -e rdiff-backup -t /var/backup

this arrangement is shown in the diagram above. additionally, this backup could be periodically pulled from another backup server using rsync over ssh. this backup server could be located on or off-site.

this could be simplified by writing the backup directly onto another server in a single xenBackup command. this is arguably less secure since you need to push the backup rather than pull it, but could be done with:

$ sudo xenBackup -a -e rdiff-backup -t root@backupserver:/var/backup

running xenBackup on multiple dom0's the following arrangement can easily be achieved:


the backups

one of the great things about xen is that the backup allows you to reconstitute a fully working xen guest from the backup area, simply with a command like:
$ sudo xen-create-image --copy=/var/backup/mymachine.rdiff-backup.mirror --ip=192.168.1.10 --hostname=mymachine

if rdiff-backup is used as the backup engine, the xen guest can easily be restored to a historical state, to as far back as increments are kept (controlled by the xenBackup purge flag, -p). see man rdiff-backup for more information on using --restore-as-of on your backup directory.

dependencies

the xenBackup script has dependencies on rsync and rdiff-backup, if you choose to use those engines. if you do, you should install those packages:
$ sudo apt-get install rsync rdiff-backup

automation

to automate your backups, consider adding a cron entry to automatically run xenBackup on your backup server. typically you should do this at a quiet time. an example cron entry is:
00 1 * * * /usr/bin/xenBackup -q -a -t /var/backup -e rdiff-backup

you should consider adding the backup server's identity file to the authorized keys of the backup user on machines to push backups to. to allow the automation of encrypted, automated backups to be pushed over ssh. read up on ssh-copy-id and ssh-keygen for more information. give very careful consideration to security when determining the user to use and what machines can access what.

a word on syslog

xenBackup logs all backup output to syslog's local3 facility. all xenBackup's log output is available in the main syslog log, /var/log/syslog. in addition, a dedicated xenBackup log can be created by adding the following to your /etc/syslog.conf file:
# xenBackup logging: log all local3's messages to /var/log/xenBackup
local3.*                        /var/log/xenBackup

if you'd like to be informed about critical backup problems by email, please refer to my earlier blog how to setup real-time email-notification for critical syslog events.

tech blog

if you found this article useful, and you are interested in other articles on linux, drupal, scaling, performance and LAMP applications, consider subscribing to my technical blog.

xenBackup - the source

here's the source for the backup script. please feel free to use, modify, plagiarize, mock, torture or hack up any of this bash code as your mood takes you.
#!/bin/bash
#
#   Copyright John Quinn, 2008
#
#   This program is free software: you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation, either version 3 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program.  If not, see <http://www.gnu.org/licenses/>.

#
# xenBackup - Backup Xen Domains
#
#             Version:    1.0:     Created:  John D Quinn, http://www.johnandcailin.com/john
#

# initialize our variables
domains="null"                           # the list of domains to backup
allDomains="null"                        # backup all domains?
targetLocation="/tmp"                    # the default backup target directory
mountPoint="/mnt/xen"                    # the mount point to use to mount disk areas
xenDiskArea="/dev/skx-vg"                # the LVM volume group to use
shutdownDomains=false                    # don't shutdown domains by default
quiet=false                              # keep the chatter down
backupEngine=tar                         # the default backup engine
rsyncExe=/usr/bin/rsync                  # rsync executable
rdiffbackupExe=/usr/bin/rdiff-backup     # rdiff-backup executable
tarExe=/usr/bin/tar                      # tar executable
xmExe=/usr/sbin/xm                       # xm executable
purgeAge="null"                          # age at which to purge increments
globalBackupResult=0                     # success status of overall job

# settings for logging (syslog)
loggerArgs=""                            # what extra arguments to the logger to use
loggerTag="xenBackup"                    # the tag for our log statements
loggerFacility="local3"                  # the syslog facility to log to

# trap user exit and cleanup
trap 'cleanup;exit 1' 1 2

cleanup()
{
   ${logDebug} "Cleaning up"
   cd / ; umount ${mountPoint}

   # restart the domain
   if test ${shutdownDomains} = "true"
   then
      ${logDebug} "Restarting domain"
      ${xmExe} create ${domain}.cfg > /dev/null
   fi
}

# function to print a usage message and bail
usageAndBail()
{
   cat << EOT
Usage: xenBackup [OPTION]...
Backup xen domains to a target area. different backup engines may be specified to
produce a tarfile, an exact mirror of the disk area or a mirror with incremental backup.

   -d      backup only the specified DOMAINs (comma seperated list)
   -t      target LOCATION for the backup e.g. /tmp or root@www.example.com:/tmp
           (not used for tar engine)
   -a      backup all domains
   -s      shutdown domains before backup (and restart them afterwards)
   -q      run in quiet mode, output still goes to syslog
   -e      backup ENGINE to use, either tar, rsync or rdiff-backup
   -p      purge increments older than TIME_SPEC. this option only applies
           to rdiff-backup, e.g. 3W for 3 weeks. see "man rdiff-backup" for
           more information

Example 1
   Backup all domains to the /tmp directgory
   $ xenBackup -a -t /tmp

Example 2
   Backup domain: "wiki" using rsync to directory /var/xenImages on machine backupServer,
   $ xenBackup -e rsync -d wiki -t root@backupServer:/var/xenImages

Example 3
   Backup domains "domainOne" and "domainTwo" using rdiff-backup purging old increments older than 5 days
   $ xenBackup -e rdiff-backup -d "domainOne, domainTwo" -p 5D

EOT

   exit 1;
}

# parse the command line arguments
while getopts p:e:qsad:t:h o
do     case "$o" in
        q)     quiet="true";;
        s)     shutdownDomains="true";;
        a)     allDomains="true";;
        d)     domains="$OPTARG";;
        t)     targetLocation="$OPTARG";;
        e)     backupEngine="$OPTARG";;
        p)     purgeAge="$OPTARG";;
        h)     usageAndBail;;
        [?])   usageAndBail
       esac
done

# if quiet don't output logging to standard error
if test ${quiet} = "false"
then
   loggerArgs="-s"
fi

# setup logging subsystem. using syslog via logger
logCritical="logger -t ${loggerTag} ${loggerArgs} -p ${loggerFacility}.crit"
logWarning="logger -t ${loggerTag} ${loggerArgs} -p ${loggerFacility}.warning"
logDebug="logger -t ${loggerTag} ${loggerArgs} -p ${loggerFacility}.debug"

# make sure only root can run our script
test $(id -u) = 0 || { ${logCritical} "This script must be run as root"; exit 1; }

# make sure that the guest manager is available
test -x ${xmExe} || { ${logCritical} "xen guest manager (${xmExe}) not found"; exit 1; }

# assemble the list of domains to backup
if test ${allDomains} = "true"
then
   domainList=`${xmExe} list | cut -f1 -d" " | egrep -v "Name|Domain-0"`
else
   # make sure we've got some domains specified
   if test "${domains}" = "null"
   then
      usageAndBail
   fi

   # create the domain list by mapping commas to spaces
   domainList=`echo ${domains} | tr -d " " | tr , " "`
fi

# function to do a "rdiff-backup" of domain
backupDomainUsingrdiff-backup ()
{
   domain=$1
   test -x ${rdiffbackupExe} || { ${logCritical} "rdiff-backup executable (${rdiffbackupExe}) not found"; exit 1; }

   if test ${quiet} = "false"
   then
      verbosity="3"
   else
      verbosity="0"
   fi

   targetSubDir=${targetLocation}/${domain}.rdiff-backup.mirror

   # make the targetSubDir if it doesn't already exist
   mkdir ${targetSubDir} > /dev/null 2>&1
   ${logDebug} "backing up domain ${domain} to ${targetSubDir} using rdiff-backup"

   # rdiff-backup to the target directory
   ${rdiffbackupExe} --verbosity ${verbosity} ${mountPoint}/ ${targetSubDir}
   backupResult=$?

   # purge old increments
   if test ${purgeAge} != "null"
   then
      # purge old increments
      ${logDebug} "purging increments older than ${purgeAge} from ${targetSubDir}"
      ${rdiffbackupExe} --verbosity ${verbosity} --force --remove-older-than ${purgeAge} ${targetSubDir}
   fi

   return ${backupResult}
}

# function to do a "rsync" backup of domain
backupDomainUsingrsync ()
{
   domain=$1
   test -x ${rsyncExe} || { ${logCritical} "rsync executable (${rsyncExe}) not found"; exit 1; }

   targetSubDir=${targetLocation}/${domain}.rsync.mirror

   # make the targetSubDir if it doesn't already exist
   mkdir ${targetSubDir} > /dev/null 2>&1
   ${logDebug} "backing up domain ${domain} to ${targetSubDir} using rsync"

   # rsync to the target directory
   ${rsyncExe} -essh -avz --delete ${mountPoint}/ ${targetSubDir}
   backupResult=$?

   return ${backupResult}
}

# function to a "tar" backup of domain
backupDomainUsingtar ()
{
   domain=$1

   # make sure we can write to the target directory
   test -w ${targetLocation} || { ${logCritical} "target directory (${targetLocation}) is not writeable"; exit 1; }

   targetFile=${targetLocation}/${domain}.`date '+%d%b%y'`.$$.tar.gz
   ${logDebug} "backing up domain ${domain} to ${targetFile} using tar"

   # tar to the target directory
   cd ${mountPoint}

   ${tarExe} pcfz ${targetFile} * > /dev/null
   backupResult=$?

   return ${backupResult}
}

# backup the specified domains
for domain in ${domainList}
do
   ${logDebug} "backing up domain: ${domain}"

   # make sure that the domain is shutdown if required
   if test ${shutdownDomains} = "true"
   then
      ${logDebug} "shutting down domain ${domain}"
      ${xmExe} shutdown -w ${domain} > /dev/null
   fi

   # unmount mount point if already mounted
   umount ${mountPoint} > /dev/null 2>&1

   # mount the xen disk read-only
   xenDisk=${xenDiskArea}/${domain}-disk
   test -r ${xenDisk} || { ${logCritical} "xen disk area not readable. are you sure that the domain \"${domain}\" exists?"; exit 1; }
   ${logDebug} "Mounting ${xenDisk} read-only"
   mount -r ${xenDisk} ${mountPoint} || { ${logCritical} "mount failed, does mount point (${mountPoint}) exist?"; exit 1; }

   # do the backup according to the chosen backup engine
   backupDomainUsing${backupEngine} ${domain}

   # make sure that the backup was successful
   if test $? -ne 0
   then
      ${logCritical} "FAILURE: error backing up domain ${domain}"
      globalBackupResult=1
   else
      ${logDebug} "SUCCESS: domain ${domain} backed up"
   fi
     
   # clean up
   cleanup;
done

if test ${globalBackupResult} -eq 0
then
   ${logDebug} "SUCCESS: backup of all domains completed successfully"
else
   ${logCritical} "FAILURE: backup completed with some failures"
fi

exit ${globalBackupResult}
bookmark/search this post with:

Cleaned up the code and

on december 29th, 2009 Justin (not verified) says:

Cleaned up the code and fixed some issues with rdiff-backup. This is now the latest version. I've posted it on pastebin:

http://pastebin.com/f3b2a5de6

I found several small issues

on december 29th, 2009 Justin (not verified) says:

I found several small issues (such as, you need to create the snapshot directory on the *remote* host, not the local host when doing a remote backup with rdiff-backup). I also added a few other things to make it work well for an rdiff-backup solution. It's now cleaned up a bit and properly tabbed too. I hope some others find this useful. Enjoy!

I've posted it here: http://pastebin.com/f3b2a5de6

Thanks John for this great,

on november 25th, 2009 Emmanuel (not verified) says:

Thanks John for this great, simple and easy to understand script !

Your work is much

on august 11th, 2009 Alessandro Fazzi (not verified) says:

Your work is much appreciated! I'm editing your script to fit my flat disk images setup easily and with very good results.

Regards,
Alessandro

thankyou for your hard work

on june 18th, 2009 darren (not verified) says:

thankyou for your hard work John.
All my domu's are flat files and stored at .img i dont use LVM.

i have a question , does the script only backup LVM based xen disks ?

yes, but it should be a

on june 19th, 2009 john says:

yes, but it should be a simple modification to use your flat files.

The LVM snapshots addition

on may 15th, 2009 Andrew (not verified) says:
The LVM snapshots addition sound just what I am looking for - but is there any documentation on how to use it?

Very impressive script! But

on january 21st, 2009 Louise (not verified) says:

Very impressive script!

But I don't understand why

xenBackup -a -e rdiff-backup -t /var/backup

is a safe back up? Won't the file system be in an unsafe state? I.e. corrupted files.

And what about databases or other programs that cache their data before writing it? That cache that is in the memory won't get backed up...

Lots of love,
Louise

Louise, I address that point

on january 22nd, 2009 john says:
Louise, I address that point in the article:

xenBackup could be used to periodically backup each of the local guests from the dom0. this is safe to do on a running server since the database backup does not rely on datafile consistency, but instead on the hot backups.

alternatively, the hot backup could be avoided if each of the guests was cleanly shutdown before the backup. xenBackup supports both these modes of operation but the former is recommended.

Hi, Very comprehensive

on january 16th, 2009 guest (not verified) says:

Hi,

Very comprehensive script. One thing I am having trouble with is what do you mean by the "xenDiskArea" var?

My domU images are in /var/lib/xen/images and I've been playing around with the var above but no luck.

Thanks for the script, its

on september 4th, 2008 Wilb (not verified) says:
Thanks for the script, its saved me a chunk of time :-)

I've made a few basic modifications to it which some others may find useful - unfortunately its not too pretty and makes an assumption that any VMs living in your volume group have a name ending "-disk" (much like xen-tools creates by default) but much like before feel free to hack it to death to make it do what you want. The functions I've added are:

  • Use of LVM snapshots. (means you can guarantee data consistency for the backup if you're doing it from a running VM)
  • NTFS remounting support (enables me to rsync files out of an NTFS Windows disk image)
  • "all" now backs up all logical volumes in the volume group with a suffix of "-disk" as opposed to just the running domUs. Not pretty but its how we have our system set up.
  • added a "dd" option to backup directly to a compressed image of a block device. Once again this one is aimed at giving me a working copy of a Windows disk image to rsync over to a DR site.

I've just finished tidying a couple of bits up in it so there is a slight chance I've just broken it, but it should be OK. You can download it here.

Nice script and nice

on september 11th, 2008 Doug (not verified) says:

Nice script and nice enhancement. One note though, after taking the LVM snapshot and before taking the backup I would restart the VM if it were shutdown.

thanks for posting this. the

on september 7th, 2008 john says:

thanks for posting this. the addition of lvm snapshots is a great idea.

Hi! Which tool did you use

on july 8th, 2008 guest (not verified) says:

Hi!

Which tool did you use to draw the diagrams above ? They really look nice.

Thanks.

visio

on august 28th, 2008 john says:

visio

Hello, this script is great

on july 3rd, 2008 weez (not verified) says:

Hello,
this script is great but iam experiencing some problems:

backing up domUs which ive created with debootstrap work well, but if i try to backup an domU which was created out of an backup i keep getting errors:

$ xenBackup -e rdiff-backup -d test -t /var/backup
xenBackup: backing up domain: test
xenBackup: Mounting /dev/lvmxen/testt-disk read-only
xenBackup: backing up domain test to /var/backup/test.rdiff-backup.mirror using rdiff-backup
Fatal Error: File /mnt/xen does not look like an increment file.

Perhaps u can give me a hint.

with kind regards

weez

weez, perhaps you could

on july 3rd, 2008 john says:

weez, perhaps you could contact me directly and let me know how you are creating your domU from backup and i'll point you in the right direction.

hi, i think i solved

on july 4th, 2008 weez (not verified) says:

hi, i think i solved it...

after creating a domU from backup with copy i forgot to remove /rdiff-backup-data/ on my new domU, now its working fine.

weez

Great script, thanks. I just

on may 13th, 2008 Marc Sutter (not verified) says:

Great script, thanks.

I just wondering why you don't use LVM snapshots but "mount -r".
Isn't the backup more consistent with snapshots, or is it just to reduce disk space needs ?

Marc

marc, to be honest, i didn't

on may 13th, 2008 john says:

marc, to be honest, i didn't consider it. it sounds like a good alternative, albeit with the possible downside that you point out.

any interested in posting a patch to add this as an option?

Great script, I was looking

on march 31st, 2008 Devon (not verified) says:

Great script, I was looking for a good way to backup my xen domains, thanks again! John!

Looks great, though it could

on march 31st, 2008 jcinacio (not verified) says:

Looks great, though it could use support for loop mounted disk images :)

Awesome script work, John!

on february 11th, 2008 Paul Lathrop (not verified) says:

Awesome script work, John! Thanks!

post new comment

the content of this field is kept private and will not be shown publicly.
  • web page addresses and e-mail addresses turn into links automatically.
  • allowed html tags: <h2> <h3> <h4> <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • lines and paragraphs break automatically.
  • you may post code using <code>...</code> (generic) or <?php ... ?> (highlighted php) tags.

more information about formatting options

captcha
are you human? we hope so.
copy the characters (respecting upper/lower case) from the image.