Backing up live servers on Citrix XenServer for free

Daily backups are still the best way to get a virtual machine back on it’s feet, and then restore them the rest of the way with CDP if you have it.

Backing up a live server in Xen is super simple, and restoring it is as well. The trick with backups is always getting them to run consistently, automatically, and verifiably.

There is a great script here from Andy Burton that I use daily to export ~20 virtual servers to a USB disk attached to one of the Xen Servers.

I have tweaked it very slightly and have some cleanup scripts to handle disk remounts, removal of older backup images, and some logic to not back up if the backup drive is not present and mounted.

audit.sh – A plaintext dump of all the info needed to figure out what used to be connected to what and where it used to live, all the SR, VM, VIF, UUID’s etc. are here in a reasonably readable format if needed.

cleanup.sh – unmounts and remounts the backup disk, and then cleans it up so that we only have the last two backups on it. Needs some logic to abort if the drive isn’t, or can’t be, mounted.

crontab.txt – My listing of jobs and order of them to run. Times are up to you.

meta-backup.sh – Backs up the metadata of the Xen Pool in a restorable format. Backs up the host machines over to the backup drive as well.

mailheader.txt – The simple header for outbound emails

and be sure to download the xenserver_backup.tar.gz script too from above .

You will need somewhere to put the backups, it can be an NFS share, SMB share, USB disk, flash drive, or anything else you can get mounted up.

DO NOT BACKUP VM’S TO THE XENSERVER “/” PARTITION! It does not have enough space to backup more than the tiniest VM and you WILL crash your XenServer and have to spin it up with a live CD and clear out whatever 2GB+ file you just accidentally made!

Note that you can back up all the VM’s, Xen hosts, and metadata from a single Xen host, so you only need to set this up on one machine. I use a KingWin USB “Toaster” style dock to keep a 320GB SATA disk in and storing daily backups, which currently is enough for two days worth. It’s a two slot toaster, so I bring in a 1.5TB disk monthly for extra copies of snapshot VM’s as well as encrypted file-by-file backups of the file servers.

Setting up an external disk(s):

fdisk -l (that’s an L)
(look for the backup drive, probably /dev/sdb but not always)

Partitioning
fdisk -l /dev/sdb     (that’s an L)
Press p to see current partitions
If there is more than one, delete it with d
select the highest number partition
keep going until they are all gone
Press n for new partition
p for Primary
Partition 1
defaults on the rest
Press w to write the changes and exit

Formatting
mkfs.ext3 /dev/sdb1
(this will take a little while for a large drive)

Make it usable in the filesystem:

mkdir /mnt/backup
mount /dev/sdb1 /mnt/backup

From here, you can export and import by hand:
**************************************
Export
(Tip! Use TAB to auto complete it all!)
[[email protected] ~]# xe vm-export         {TAB}{TAB}
filename=              preserve-power-state=  vm=
[[email protected] ~]# xe vm-export vm=  {[TAB][TAB]}
{A list of servers appears, We want to export Pokey Server}
[[email protected] ~]# xe vm-export vm=Pokey\ Server  {Type in Pokey then [TAB][TAB]}
filename=              preserve-power-state=  vm=
[[email protected] ~]# xe vm-export vm=Pokey\ Server filename=
[[email protected] ~]# xe vm-export vm=Pokey\ Server filename=/mnt/backup/pokey.xva {Type in /mnt/backup/pokey.xva then press ENTER}

After a bit, the pokey machine is backed up onto the external hard disk. This is a fairly quick operation, usually far faster than a file-by-file backup (500MB/minute on average) and note that it compresses the backup on the fly.

Setting up the automatic backups:
***************************************

From a XenServer with a USB drive attached:

cd /root
wget http://www.8layer8.com/xen/audit.sh
wget http://www.8layer8.com/xen/cleanup.sh
wget http://www.8layer8.com/xen/crontab.txt
wget http://www.8layer8.com/xen/dbtool
wget http://www.8layer8.com/xen/meta-backup.sh
wget http://www.8layer8.com/xen/xenserver_backup.tar.gz

chmod +x *.sh
chmod +x dbutil
cp ./dbutil /sbin/dbutil
chmod +x /sbin/dbutil
tar -zxvf xenserver_backup.tar.gz

nano meta-backup.sh
(change the names and filenames to the names you have for your XenServers)
Control-X to exit, answer Y to save changes

nano vm_backup.cfg

Edit the log_path to be “/mnt/backup/vm_backup.log”
Edit backup_dir to be “/mnt/backup”
Edit backup_vms to be “all”

Enable email on the XenServer:

nano /etc/ssmtp/ssmtp.conf
Change mailhub=mail to mailhub=(your mail server goes here)
Change rewriteDomain=yourdomain.com

Save and Exit

Schedule the backups:
Note the times here are reverse military, so 0 19 * * *  means run at 19:00 every day, change them if you want, just keep them sequential top to bottom time-wise so the scripts run in the right order.

crontab -e
(Press i to insert text)
Paste in: (be sure to use your email address!)
MAILTO=”[email protected]
0 19 * * * /root/cleanup.sh
10 19 * * * /root/audit.sh
11 19 * * * /root/meta-backup.sh
0 20 * * * /root/vm_backup.sh

Then press Escape
Type in :wq [enter]

Done.

Check the backups tomorrow and for several days to make sure that they rotate properly.

Restoring from a backup is very simple:

Locate the backup you want to restore from, probably in /mnt/backup/thisserver_09_25_2010.xva

Locate the storage repository you want to restore it into: run xe sr-list, and find the storage you need to use, note the first 4 characters of the UUID (ex: 28f2)

Run xe vm-import like this:

xe vm-import filename=/mnt/backup/thisserver_09_25_2010.xva sr-uuid=28f25ea1-4c49-5346-4a86-d37560bd07b7 [ENTER]

This will immediately start importing the snapshotted machine into the Xen pool as thisserver_snapshot. You will want to change network connections or other means of keeping this machine and it’s evil twin on the network at the same time! If the original is dead or missing, then there’s no problem.

Occasionally, a machine may require that you add a force=true at the end of the vm-import:

xe vm-import filename=/mnt/backup/thisserver_09_25_2010.xva sr-uuid=28f25ea1-4c49-5346-4a86-d37560bd07b7 force=true [ENTER]

It would be wise to force a chkdsk or fsck on a forced import.

39 comments on “Backing up live servers on Citrix XenServer for free
  1. Rob says:

    Great script – thanks for sharing this. I’ve configured it to backup 30 VMs in a pool to a NAS and it seems to be working quite well. I like the email alerts although after I finish testing the existing configuration, I might consolidate them so I don’t get a bunch of emails each time the jobs run.

  2. Brad says:

    I split them up on purpose so that if the cleanup/mount/remount fails I see it early enough that I can intervene and restart the backups from home, but, yes, there’s no reason they cannot be consolidated. Glad you like it! Props to the originator of the backup scripts too!

  3. Gabriele says:

    WOW, Great scripts! i’ve tested all and are ok! many thank’s

  4. Nicolas says:

    Hi,

    I’m a new sys admin in a company and the previous guy in my position had setup a xenserver with several virtual machines.

    I wan to perform a back up by exporting Virtual machines on an external USB drive.

    Running fdsik -l I see :

    [[email protected] nicolas]# fdisk -l

    Disk /dev/sda: 1932.7 GB, 1932735283200 bytes
    255 heads, 63 sectors/track, 234975 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 499 4008186 83 Linux
    /dev/sda2 500 998 4008217+ 83 Linux
    /dev/sda3 999 234975 1879420252+ 83 Linux

    Disk /dev/sdb: 66.5 GB, 66571993088 bytes
    255 heads, 63 sectors/track, 8093 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System

    Disk /dev/sdc: 3999.6 GB, 3999688294400 bytes
    255 heads, 63 sectors/track, 486267 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sdc1 1 246051 1976404626 83 Linux
    /dev/sdc2 246052 486267 1929535020 83 Linux

    The problem is I don’t see any changes in the output of this command before or after attaching the external hard drive, not even in dmesg output. I have formated the hard drive with ext2 file system as I thought ntfs was not supported.

    Is there any problem with xenserver linux distribution? how can I make the xen server recognize when I attach the external hard drive so I can export my back ups ?

    PS: I don’t want to do the export to a network mount.

  5. Brad says:

    If you don’t see anything in dmesg or fdisk -l, then something odd is going on at the usb layer. You should be able to see it and format it right from the XenServer console, so if its not there, there’s something wacky with the drive or the usb port or the drive needs external power.

  6. Tim says:

    Hi,

    I’m new in using XenServer and your script is really helpful. But can i be shure, that the Snapshoot-Backup is consistent everytime?
    What about backup VMs using rsnapshot, would this work too? thank’s

  7. ebenvios says:

    Excellent post. I used to be checking continuously this weblog and I’m impressed! Extremely helpful info particularly the last phase 🙂 I deal with such info a lot. I was looking for this certain info for a long time. Thank you and best of luck.

  8. naturally like your website but you have to test the spelling on several of your posts. A number of them are rife with spelling issues and I to find it very bothersome to inform the reality nevertheless I will certainly come back again.

  9. Nuno Pratas says:

    Hi,

    I’am testing this solution and I run the scripts manual and are ok.
    Where can I find dbutil mentioned in your setup?

    Thanks

  10. Brad says:

    Hi Nuno , you can find the utility here : http://www.8layer8.com/xen/dbtool

  11. Nuno Pratas says:

    Thanks Brad!

    So when you say:
    chmod +x dbutil
    cp ./dbutil /sbin/dbutil
    chmod +x /sbin/dbutil

    it should be:
    chmod +x dbtool
    cp ./dbutil /sbin/dbtool
    chmod +x /sbin/dbtool

    Right?

    By the way, nice job!

  12. Nuno Pratas says:

    Sorry,
    Correction.

    it should be:
    chmod +x dbtool
    cp ./dbtool /sbin/dbtool
    chmod +x /sbin/dbtool

  13. Brad says:

    That should do it. I’ll check the post for correctness. Thanks!

  14. Christopher Barnett says:

    I posted this on Andy’s thread but it’s older than yours and I don’t know if I’ll get any feedback. I like the tweaks that you wrote, nice additions!

    I have a question about using another iSCSI SR for the backup target. Is this possible with a configured iSCSI SR or would I need to mount it as you have the USB drive to get the same results? I don’t want to pay for a commercial fix for this and I don’t have the wiggle room between our two devices to go the Openfiler iSCSI front end route.

    I’d love to hear your thoughts if you have the time!

    Thanks!

    -Chris

  15. Brad says:

    I don’t see why it wouldn’t work, but I would probably recommend against using that backup SR as primary storage for VM’s. If you were running in a clustered environment, there would be issues if you happened to collide on a backup name for some reason, and I’m not entirely sure that it would play nice as a general filesystem. BUT, for NFS mounts it really should be ok. For iSCSI, the XenServer should be handling it as a clustered filesystem since it’s a mounted block device, so it should behave even though it is outside of the Xen bubble. Anyway, there’s no reason not to give it a shot!

    The SR’s get mounted at /var/run/sr-mount/(GUID-OF-SR)/, so you should be able to point the backup location to that instead of a USB drive.

    Good luck!
    Brad

  16. Kima says:

    Thank you for the script! Works perfectly!
    Now.. is it possible to image only the system-harddrive or to exclude some harddisks from imaging?

    Thanks…

  17. mike says:

    Hello,

    Thank you for this great work!!

    i tested it witch xenserver 5.6 and it works fine !

    but i wanted to know if it would work witch xenserver 5.0, 5.5 and xenserver 6.0 ??

    thank you in advance 😉

  18. Brad says:

    It should work on 5.0,5.5, and 5.6. I haven’t tested it on 6.0, but it will probably work or need a minor tweak here and there.

  19. Yeah, works great on XenServer 6.1. I set it up this past month and have been running it with no problems.

    I set my mount point to a iSCSI location over a nfs share.

    Wrote up an article on my steps using a lot of what is here, but with slight alterations.

    Thanks for the nice work Brad!

  20. Steven Hammond says:

    does the dbtool work with 6.1? when I run it from the command line (dbtool -a /var/xapi/state.db) I get the following message over and over and then follows the NIC information.

    no table found

  21. Brad says:

    There may be a newer version that will support 6.1. I don’t have access to a 6.1 server to test on, but the original dbtool was located on Citrix’s site.

  22. Dominic says:

    Thanks a lot for the script. I had one problem. I didn’t see any text in the E-Mail, that was sent from the Meta-Backup-Script. In my opinion, there have to be a blank line between the subject and the body.

    So I inserted a line
    echo -e >> message.tmp
    after the line
    echo “Subject ……”

    Perhaps this hint helps other people, too.

  23. Raboo says:

    dbtool appears to be designed for 4.x and 5.x
    http://support.citrix.com/article/CTX121564

  24. Andi says:

    Thanks a lot for this script.

    But one important question i cannot answer for sure while using google for that: is a snapshot like it is created here the same as a full backup? or do i need something else from the host if the discs of the host crashes total?

  25. Brad says:

    To restore from this snapshot just needs another xenserver and then import this image as a new vm. It will likely need to run fsck or chkdsk because it wasn’t shut down cleanly and is being restored to a point in time. If the networking is different, you may need to reassign the Nic, but that’s usually about it. There’s an import/export post on my site too that is effectively the restore that you are looking for. I’ll find it and post the link.

  26. Brad says:

    http://www.8layer8.com/?p=200 is the post you need. You just need to adjust for where your backup image is accessible from and off you go!

  27. saeed says:

    i want backuping my vms on xenserver host lively,but i unknow a free or cracked softwre that operate this work. i want a free solution for my citrix xenservr backup. …help me…

  28. Brad says:

    Well, this is a free way of backing up your vms. Please read and follow the instructions in the post, and you should be able to back up the machines for free.

  29. Thamaraj says:

    Hi Brad and everyone,

    Thanks for great solution to backup Xenserver VM.
    Before I implement this backup solution on my Xenserver I just want to check whether this script will work in Xenserver 6.2 ?
    Please advise and waiting for your reply…..

  30. Brad says:

    Yes, this script is currently working under 6.2. I have customers that are using it with 6.2 daily.

  31. John says:

    –2014-09-17 13:57:52– http://www.8layer8.com/xen/audit.sh
    Resolving http://www.8layer8.com... failed: Temporary failure in name resolution.
    wget: unable to resolve host address `www.8layer8.com’

    Can’t connect to download.

    Thanks

  32. Brad says:

    Should work now, had a brief power outage.

  33. John says:

    –2014-10-19 21:53:03– http://www.8layer8.com/xen/audit.sh
    Resolving http://www.8layer8.com... failed: Temporary failure in name resolution.
    wget: unable to resolve host address `www.8layer8.com’

    Can’t connect again to 8layer8.com – power outage again

  34. Brad says:

    Not sure why you’re getting this. DNS is fronted by Cloudflare and really shouldn’t be down. I have no alerts that it is down, and am on it right now. Are you running the script from my site? The wget should pull it down and keep it. Running it directly from the site seems like a bad idea security wise.
    Xen servers may not have the best internal networking for themselves, I’ve seen more than one use no DNS at all, or point to itself or a VM that it hosts for DNS services. Either way, you should pull the scripts once and be done with that, there’s no updates happening on them any time in the future. Thanks! Brad

  35. Hesan Yousif says:

    Hi Brad,

    Weirdly, getting a Failed to copy disk. SR_BACKEND_FAILURE_44 coming up when the code tries to back up certain VM’s when there is lots of space available on the SR.

    We run XenServer 6.2 and I know there were problems with local snapshosts, but the script I have used from you has been adapted to dump all copies on a NAS device – which is also mapped as a SR within the host.

    Any ideas?
    Cheers,
    H

  36. Brad says:

    How is the NAS mounted? I’ve had occasional weirdness with NFS mounts, and depending on the backup script, I’ve unmounted and remounted the shares between VM backups. If you do that, make sure there’s a check in the script to make sure that the mount was successful so you don’t fill up the / partition if the mount doesn’t come back.

  37. Gene Cooper says:

    With XenServer 6.5, the 32-bit dbtool no longer works.

    Does anyone have a suggested replacement tool or command for the audit.sh script?

    Thanks for this article,

    G

  38. Andi Topp says:

    We got multiple “no table found” messages when running the dbtool – i can not find any information about what that mean!
    can somebody help out?

  39. Brad says:

    You might want to try and get a newer version of the tool from Citrix. I have not had a live XenServer in a long time, so I won’t be able to help you further, sorry.

2 Pings/Trackbacks for "Backing up live servers on Citrix XenServer for free"
  1. […] I chose to update/rewrite a lot of what was originally done here by Andy Burton. It was then updated and modified by Brad at Layer 8 here. […]

  2. […] I chose to update/rewrite a lot of what was originally done here by Andy Burton. It was then updated and modified by Brad at Layer 8 here. […]

Leave a Reply

Your email address will not be published. Required fields are marked *

*