VMX Bandits

Here's an interesting blog post by someone who ditched Xen in favour of ESXi, "the smaller and somewhat crippled cousin of ESX".

 Annoying, is the right word for this process. What it should involve is simply editing the settings of the VM in question, changing the size of the disk via the already provided UI widget, then hitting okay. Of course, you still need your OS or third party tool to resize the partition inside the virtual disk, but that should take care of the actual virtual disk sizing. Hey, it's not even too much of a stretch of the imagination to suggest you should be able to take a snapshot before attempting the resize so you can rollback if it fails.

But no. The UI widget does nothing. It looks cool, but making any changes results in precisely zero happening. It just reverts back to the original value when you click "ok". If you really want to resize the disk, you need to log onto the ESX server hosting the virtual disk and use the "vmkfstools" utility. So resizing a virtual disk from 4GB to 8GB is achieved like so: vmkfstools -X 8G virtualdisk.vmdk

Except that there's a few problems with this. Firstly, if your vm has any snapshots, kiss them goodbye, because you can't shrink the disk once it's explanded, and the snapshots now no longer match the underlying disk. Secondly, if you actually have any snapshots you can't remove them from inside the Virtual Infrastructure Client once you've resized the disk - it just errors out. Thirdly, it's likely that vmkfstools will not correctly resize the geometry parameters of the disk.

So how I fixed this was first of all, I manually edited the vmx file to remove the pointer to the incompatible snapshot. This involved changing the line that read "scsi0:0.fileName = "virtualdisk-000001.vmdk"" to "scsi0:0.fileName = "virtualdisk.vmdk"". I then deleted the snapshot files manually (the .vmdk and -flat.vmdk files associated with the snapshot.)

I then tried to get my partition utility to stretch the partition to fill the entire volume. Except it couldn't see past the existing size. I tried booting windows, and windows was able to see the full size and add another partition in the free space, but sadly couldn't stretch the active partition to fill the whole disk. [ I think Vista can do this, but probably not in 4GB of space ;) ]

Anyway, long story slightly shorter, turns out vmkfstools didn't adjust the geometry of the new disk correctly - the "cylinders" parameters were still on the old value of 522, not the new correct value of 1044. The specific lines were 'ddb.geometry.biosCylinders = "522"' and 'ddb.geometry.cylinders = "522"'. So then I was able to expand the partition, and everything was happy again.

Merry Christmas from all at VMX Bandits!

In my last post, I stated my intention to look into rdiff-backup as a possible method to emulate the really good space efficiency we got with ZFS over NFS forming our vmware data storage layer due to ZFS snapshots and its copy-on-write behavior. Put simply, a daily snapshot of a VM with a ~10GB virtual disk only "cost" about 100MB or so on average (representing the blocks that had changed in the virtual disk during that snapshot period.) I was looking to either match or beat that level of space efficiency with daily backups.

So, what did I do? I took two fullvm type backups of a virtual machine with an 8GB virtual disk that I'd made about two weeks apart. I took an rdiff-backup of the first one, then I copied the files from the second one over the top of the first one, and took another rdiff-backup. One thing to note is that rdiff-backup takes ages to run! When I tried to run the binary using a full name, it spat some stuff back about not finding python, so I guess that explains why it's so slow

The results? Well, bearing in mind that vcbMounter by default trims off the unused portions of the virtual disk, the actual amount of data comprising a fullvm backup is around 3.97GB (4267MB). The final size of the rdiff-backup store directory was about 4.28GB (4602MB.) This is a bit of a problem. I know I'm comparing two weeks worth of changes with a single day worth of changes, however this VM is doing practically nothing (DHCP, NTP, and a few other minor network services) and I am completely sure that the 100MB or so of changes we were originally seeing each day was in fact the same 100MB changing (logs, data files, counters, etc.) So the fact I'm seeing over three times as much data for differentials with rdiff-backup is a little annoying.

 I haven't tested any of the above with fullvm backups taken with vcbMounter's -F option, to see if it help's rdiff-backup's algorithm find more efficient diffs. Perhaps the flattening of the virtual disk to a compact virtual hard drive is responsible for causing more diffs than would otherwise exist. As a final note, I tried compressing the rdiff-backup store folder after it had both backups in it ended up the same size as either of the previous independently compressed backups, plus the ~300-400mb of diffs I mentioned above. Which I guess is what you'd expect.

So the pros and cons. Well, in favour of rdiff-backup, it does indeed seem to detect (like ZFS does) that hardly anything changes between backups, and gives it a red hot go at trying to eliminate the duplicated space. It also seems that getting back the most recent copy of the backup is easy - just copy it out: it's there on-disk as you'd see it normally. If you set up some sort of 7-zip compression on the current copy, you'd also end up with something similar to ZFS filesystem compression too (though I suspect this might actually work out to be better compression than ZFS achieves.) And of course, lets not forget that theoretically, VCB creates truly consistent images that with the help of custom freeze and thaw scripts can potentially back up anything even at an application layer completely safely.

Now for the cons. The biggest problem is of course that it's not a single well defined command like zfs snapshot is. You need to set up scripts to locate vms, export them, run the rdiff-backup's on them, potentially 7-zip and un-7-zip files before and after. Plus there's all the error handling and rollback and subtle combinations of those that go with the above sequences. The second biggest problem is zfs snapshot might take a few seconds to back up all your vms, whereas the rdiff-backup process can take literally hours for a single VM with 4GB of data.  All this for something that at best is roughly as good regarding space efficiency.

So my conclusion is that rdiff-backup may work for you if you're short on disk space, have a small vmware installation, and don't mind having to spend maybe a few hours to restore a virtual machine. For everyone else, keep praying that deduplication routines for ZFS, or some mechanism to overwrite partial diffs in-place, are created