batting cleanup when the hard drive with AD dies

subtitled: **and why NTDSutil is my new best friend**

Ever inherit a customer who’s systems were poorly designed and built? 🙂  Sorry I had to ask.  Of course many professional technology service companies have.  Unfortunately there are plenty of enterprising tech’s who have no clue what they are doing but are only happy to spend the money of a business to pursue their education on a company’s production equipment.

16 hours later and a very short night I’ve realized the importance of a System State backup, RAID, and taking a more urgent audit of all new customers’ critical infrastructure.  Also did I mention that NTDSUTIL is your friend if ever you need it.

What I ran across was a single failing IDE drive which appeared to only be used for DATA.  The OS, Windows 2003 Server Standard, was running on a mirrored set (off MB) of SATA drives configured Dynamic (God knows why as the mirror was on the hardware) and its drive letter was C:.  The D: & F: partitioned drive seemed solely used for archived information.

This customer reported the overall network to be running slow, email to be erratic, and after an initial examination of the DC which held all FSMO roles there were some files and folders that were disappearing, unreadable, or both.  The naturally worst case assumption was a security issue/breach which was a complete false lead.  A quick check of the event logs indicated “bad block” alerts on a HD.

This puzzle of purported performance pestilence had me perplexed.  So a quick assessment was made of the contents of the failing drive and what to do to salvage any data.  A mistake was made under fire to not detect that the Exchange Server’s database and logs were moved to this HD and also that AD’s database and logs had been moved there too.  Why?  well…  Its not normal to do this as it weakens the survivability of this critical keystone of the business; additionally, with time as a concern of what we could still save before eminent failure occurred this just wasn’t something we had foreseen to look for.

With the limping HD still connected the unorthodox Exchange configuration was fixed by moving the database and log for all stores to the lone good logical drive where afterwards 1% free space remained (it was tight but it fit – whew!).  When the bad HD was removed and system rebooted you might anticipate that I quickly learned that AD was unavailable and that we had some serious problems.  The error message informed me to boot into Directory Service Restore Mode (DSRM).  The question was why and what to do when we get booted there.  However after some auditing and a few very sluggish reboots cycles with the bad HD we discovered where AD resided.

A reboot into DSRM allowed the running of the NTDSutil windows command tool which had a provision for moving the database and logs.  The database moved to the good HD but the logs didn’t.  OH NO (cursing ensued)!  Now what?  Tried a few more things with the NTDSutil tool but didn’t have any luck.  Tried rebooting hoping new logs would be rebuilt but that wasn’t happening.  Called Microsoft midnight India support and abandoned that avenue when suggestions of dcpromo demotion, registry hacking, and Exchange reconfiguration craziness felt very close to disastrously dangerous to even attempt.  It was discovered at this moment of inspiration that another untried command of NTDSutil might work; we specified the location of the log files not where they were but rather where we wanted them to move.  The log files wouldn’t be there but perhaps AD would rebuild them on boot up.  Eureka that was the fix!

The cleanup was to add a new mirrored set of much larger HD’s and move the Exchange mailstore database and logs.  Also some of the recovered data was placed back.  The System drive had 15% free space, all worked well enough, and the business would be unaffected at the start of the day.   With a feeling of victory and satisfaction, off we went get home for some R&R as dawn approached.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s