Lessons Learned–HyperV go BOOM

Old client called today with a really good problem.  This happens.

This time is was a really really good problem.

Boom – I mean BOOM – HyperV (Win2k8R2) VM went down and this guy was flustered and seeking some solace.  Why?  A virtual disk (dynamic) filled up the physical space it was placed inside and ran out of room to keep dynamically growing.  Why?  poor design in allocating physical drive space aka it was over-committed (the old adage about packing 4lb’s of excrement into a 2lb bag comes to mind)

First Lesson – a VM will ‘pause’ automatically if every this condition is met.

First Lesson Pt B – defragmenting a VM is dumb for many reasons …in this case it took a small dynamically expanding hard drive and inflated it as it performed reads on every block to determine what was/wasn’t fragmented data

First Lesson Pt C – expansion is one directional – expanded virtual HD’s don’t auto deflate …ever ever ever

Second Lesson – physical limits force all spacial designs – a HD space exceeded genie might be able to go back into the bottle …a little bit.  What I mean is that you can actually ‘shrink’ a Windows partition if you have unused contiguous space inside it (well the defragmentation helped in this way at least – wasn’t all bad)

Second Lesson Pt B – even if you can shrink your vHD, the physical file is the same damn size – duh! (more to do)  To subsequently shrink the space inside the VM you will have to run a boot CD partition manager aka Gparted …etc or try the free AOMEI Partition Assistant which run inside Windows

Third Lesson – you really ultimately need to ‘Compact’ your .vhd and so subsequently shrink its footprint in the physical partition it is sitting in on the Parent hypervisor OS.

Third Lesson Pt B – you can’t compact a vHD if there are ANY shadow copies on the partition inside it.  There are a lot of Google results pointing to the errors you will get if you try and fail.  In the end you need to delete all ‘Shadows’ – vssadmin delete shadows /all (elevated CMD)

Third Lesson Pt C – If you have a third party backup software that leverages the VSS writers/engine …ensure it is disabled.  I had StorageCraft’s ShadowProtect kicking off a backup once the VM started up each time and consequently created a new ‘Shadow’.  This resulted in more failure.  This one wasn’t found through Google and it took me a little to realize and resolve this last obstacle.

Fourth Lesson – Actually one of my first steps was to move the smaller C:\ drive’s fixed vHD file off to an external drive.  The goal was to start the VM up so to gracefully shut it down.  Later to find and remove the ‘Shadows’, disable the ShadowProtect backup jobs …etc.  I moved the file over to a large 3TB format drive.  In the end once all was good and small with the other vHD then I moved it back.

Fourth Lesson Pt B. – you can’t boot a VM when one of its vHD’s is on one of these big whopper drives …well at least as natively formatted.  Why?  The underlying sector sizes need to use 512 Bytes

Fourth Lesson Pt C – How do you find out what your disks geometry is?  Run a nifty utility:  fsutil fsinfo ntfsinfo X: (where X is the drive letter of the target) elevated command prompt



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s