SBS 2011 failed restore – KSOD black screen with cursor

Crisis compounded …a business is critically impacted and NOW like as in NOW need to use their Windows Backup to restore back to a known good point.  You go through the 1-2 hour process and when the server reboots you get …a Black Screen of Death (KSOD) aka black screen a cursor and an unhealthy server OS that you cannot access.

When I hit this issue I immediately called a colleague who laser beamed onto the source of this issue.  Even with that information of the general cause and fix in hand, I had a very difficult time finding the needed detailed information so I could apply it and quickly overcome the issue.  My client needed to get back up and in business – my ass was now on the line!

…So I’m rectifying this info sprawl disconnect and putting it all in one place  – here.

In the interest of helping other businesses get back up from this scenario as fast as possible, below is this assemblage blog post.  This assembled information resolved my issue and can resolve yours as well.  It really isn’t very complicated to execute and you will find yourself quickly de-stressed once you again see the Ctrl-Alt-Delete screen following the server’s startup.

This also can apply to SBS 2008 too; just remember to copy the directory mentioned below from an SBS 2008 source.

First – What, Why, When

• The reason you aren’t fully booting into the OS GUI is due to a previously installed Microsoft tool that you will later want to remove. This tool is the NTbackup & Restore Tool.  It is installed as a individually downloadable KB (.msu) which means it doesn’t appear as a standalone Role, Feature, or Program.  Unlike those things which you are commonly used to manipulating this tool has to be removed via the “Installed Updates” list …more on that later.
• This Microsoft Tool is interfering with a specific directory that is now incomplete and without it your recovery fails.  You end up at a KSOD (Black Screen of Death).

Second – Help!

• The ‘kinda’ good news is that if you can get a copy of this directory from a known good installation of the same OS at another of your clients or from someone you trust to provide you the same …then you are ‘in like Flynn’

The reason for this is that a specific folder is omitted from the backups – c:\Windows\Registration – if this condition exists.

This condition is now noted in Update Rollup 3 for SBS 2011 – http://blogs.technet.com/b/sbs/archive/2012/06/13/update-rollup-3-for-wssg-bpa-is-now-available-via-microsoft-updates-along-with-its-installation-tool.aspx

The simple fix for this is as follows:

1. Copy the contents of C:\Windows\Registration from a working SBS2011 server
2. Boot into the Windows Repair/Recovery mode using SBS2011 Disc 1
3. Open up a command prompt and browse to C:\Windows\Registration
4. Copy the files (from a USB drive or something) over to the server

The server should now boot up

(see reference link to PowerBiz Solutions post by Boon Tee below)

Third – Cleanup

• once the OS is back up and running you’ve got some critically important work to do
• uninstall the NTbackup & Restore Tool by bringing up either Programs and Feature or the Widows Updates Windows and then selecting the ‘Installed Updates’ choice on the left column.  Within the list of updates find and remove
  “Update for Microsoft Windows Windows (KB974674)”
• install, review, and take remediation steps from the Microsoft Windows Server Solutions BPA 1.3 – avoid aka ignore the direction to alter the webconfig as instructed with the EWS error (see references below for further information)
• run a full image backup immediately

(click above to enlarge)

References:

• SBS 2011 Build Wiki – V. Backup, Restore, and Disaster Recoveryhttp://social.technet.microsoft.com/wiki/contents/articles/1919.v-backup-restore-and-disaster-recovery.aspx
• Susan Bradley – remove that http://msmvps.com/blogs/bradley/archive/2011/12/25/remove-kb974674.aspx
• problem discussion in Microsoft Forums http://social.technet.microsoft.com/Forums/en-US/smallbusinessserver/thread/be28b0a1-9cad-45aa-ad4c-b9e85e3128d2/?prof=required
• the fix – Fixing the Black Screen of Death (KSOD) on SBS 2011
  http://blog.powerbiz.net.au/fixes/fixing-the-black-screen-of-death-ksod-on-sbs-2011/
• BTW – don’t do that with regards to the latest Windows Server Solutions BPA 1.3 – EWS can be broken
  http://blogs.technet.com/b/sbs/archive/2012/06/13/update-rollup-3-for-wssg-bpa-is-now-available-via-microsoft-updates-along-with-its-installation-tool.aspx?wa=wsignin1.0&CommentPosted=true#commentmessage

SBS gone poof …Transition Ahead! – Who, What, Why, Where, When

subtitle –  emerging technology innovation train coming through – this just isn’t the right business for stagnation

News is now out …SBS is no more – dramatic shift and what exactly does this mean?  …losing the name SBS, the unique partner certification and designation branding SBSC, and much of the special philosophy that the last decade saw Microsoft reaching for the Small Business as had never happened before. 

Who – the customer, the partner, the vendors, the software developers, the software support …all will be impacted

What – SBS the name and the brand is no more, no SBSC, replaced by a hybrid of Windows Server Essentials 2012 & either a member server of Exchange on premise or Exchange hosted – all the predictable results wizards gone  …it’s a new world

Why – …apparently it was time for a change – life happens …business decisions are made by those that shoulder the rudder and compass

Where – all encompassing across the width and breadth of Microsoft’s landscape

When – SBS 2011 w/ SA stops being sold July 31’st, 2012, SBS 2011 platform stops as OEM December 31’st, 2013, SBS 2011 platform in VL and … stops June 30, 2013 – the Windows Server Essentials 2012 product launches with the Windows 2012 family of server expected in the August-September time frame.

---

**other important noteworthy aspects**

none of the new licensing and pricing for this model is out yet -  it is not yet possible to do an accurate cost comparison of whether buying SBS 2011 w/ SA now will be a good decision.  The historical Microsoft trend has been to supply the full next version products and licensing to be at a minimum equivalent to what SA covers; in the case of now defunct Windows Essential Business Server aka WEBS aka EBS, that meant a really incredibly good deal.  With the SBS 2012 platform that will likely mean Exchange, Exchange CALS, and Windows Server Standard 2012 …see the linked FAQ from the preceding post.

a script is under development to help integrate on-premise Exchange on a member server to Windows Server Essentials 2012 so that managing users and their email accounts can be done from the WSE console …sorta SBS-like – this integration script is available in the current downloadable beta http://www.microsoft.com/en-us/download/details.aspx?id=30327

Windows Server Essentials 2012 will be the evolutionary step product from what is currently named Windows SBS 2011 Essentials and the licensing model of 25 users and no CALS required should carry forward.  WSE 21012 will add Direct Access as a new feature.

WSE will be able to be ‘transmogrified’ aka 25 user limit breakable via a one time purchasable upgrade script.  There will be no unique upper limit as the server essentially become a Windows Standard Server OS.  The de-duplication backup feature will expand up to 75 devices at this point.  Once the ‘transmog’ is applied you still get to keep the RWA, Remote Web Access, and De-dup Backup features.

SBSC is gone …Small Business competency (Silver or Gold) is an available path to consider going.  The benefits are in many ways nicer.  It costs nearly 562% more ($1,850 Silver versus $329 MAPS – as of today for US Partner)  https://mspartner.microsoft.com/en/us/pages/membership/small-business-competency.aspx

---

**General Banter**

Ouch that hurt …didn’t see that coming?  Probably because up to a month ago their were reports of development on a product to replace SBS 2011 still underway.

Although the never-ending siren call of ‘go to the cloud’ beckons, many on-premise clients are resisting; their LoB apps, data security requirements, and their bandwidth costs are key factors.  Resistance is futile or so I’ve been told.

So while you can still do on-premise, it isn’t for those that are simpletons or light on their ability to follow complex instructions and do their due diligent preparations to ensure good healthy rollout projects.  We’ve been encouraged and guided to adapt to the SBS mantra of putting everything possible on a single box to maximize value to the customer.  In this era, we can now still do this but can silo the various resource monster products onto their own virtual machine where all the products like Exchange, IIS, and SQL can coexist but yet be constrained by assignment of memory or CPU cores.

---

**Silver Lining**

As painful as this transition is, if we can get all the keystone IT infrastructure software components in VM’s under a single Virtual Parent and keep pricing in line with its historical small business affordability; then we will be in a better place.  Having a BDC and redundant DNS is in reach.  Another perk is that ISV’s can design their LoB software for Windows Server and not have to specially accommodate a ‘small’ flavored unique server platform as has been the case.  This also provides the ability to opt into increased DR resiliency by implementing the usage of HyperV clustering with the VM’s on a common NAS.  For those business that never want to be down for long; this is a great new option albeit with a doubled++ price on hardware.

autorun.inf AV blocking gotcha

Discovered this today from within Windows 7. …Any software process that involves copying, moving, or even deleting the autorun.inf file can fail as a result of current default antivirus software (aka AV) behavior.  This file is at the root of every Windows drive.

In preparation for an SBS 2011 install I wanted to create a bootable USB drive of the install media.  To start I attempted to clean off the drive of all its contents but that process failed with the autorun.inf file.  Then I realized that I should’ve just reformatted so to be thorough; of course that sledge-a-matic action worked.

I’ve done this before for SBS 2011 and WHS 2011 which can require this type of USB drive install media on headless servers.  The odd thing is that never before was this an issue so this must be something new that has emerged behind the security scenes but of which I was not aware.  What I uncovered was the Trend Micro WFBS Agent settings were blocking both ‘delete’ and ‘copy’ actions to the autorun.inf file.  This isn’t specific to just TM though as the Google search result I found pointed to a different AV vendor.  If you hit this error, disable the AV temporarily as the workaround. 

Initially I opted to go the Windows 7 USB/DVD Download tool method (creates a bootable USB drive from ISO or DVD) but hit an issue when it failed during the copying process.  After trying variations and getting the same failed result, I opted to go the longer manual creation route as detailed by Tim Barrett in his www.NoGeekLeftBehind.com blog.  During that process I hit the root issue in a way that gave me a usable error message to find the solution.  After disabling the AV I hit success.

RWA connections from XP for SBS 2011, WHS 2011, SBSE 2011, WESS 2011

There is an issue when attempting remote connections from Windows XP SP3 through SBS 2008 or SBS 2011 as both use a Remote Gateway implementation that requires a couple of updates. To succeed at reaching your remote office desktop from your home or offsite based XP desktop (not applicable to Vista or later OS) you need to install two things to make this work:

Here are the links:
· http://www.microsoft.com/download/en/details.aspx?id=20609 – XP RDP 7.0 client – doesn’t require restart

· http://support.microsoft.com/kb/951608  Microsoft FixIT – repairs Credential Security Support Provider (CredSSP) Service in XP – requires a restart

**ref** SBS 2011 release documentation which covers the second linked fix above and other possible remote connection issues:

http://technet.microsoft.com/en-us/library/gg491249.aspx

SPLwow64.exe Terminal Server/Remote Desktop Service tweak needed

Now having had two clients with a related system process problem, I am documenting what I’ve discovered and the needed tweak to fix it.  Both of these small businesses are heavily using Remote Desktop Services aka Terminal Server.  One with SBS 2011 and Windows 2008 R2 on the member server (both virtualized on Hyper-V); the other uses SBS 2008 with Windows 2008 on its member server.  The first biz uses 98% of its connections as thin clients and has seen huge amounts of memory (commit size) set aside for instances of ‘splwow64.exe’.  The second client experienced an issue with Windows 2000 clients (yes they are still out there …yikes!) not automatically ending their session when they closed the Environment Tab specified application that their TS session is limited to running/displaying – just got the blue logoff screen in a hung stasis.

On the first I tried changing the configuration of the main shared printer.  The printer is which I determined was where all the SPLwow64.exe related print tasks were being sent.  First I disabled spooling and secondly I unchecked the box to render print jobs on client (in this case the RDS server).  Neither satisfied the desired memory release I wanted to see achieved.  I could confirm the correlation with the SPLwow64.exe process and actual memory consumed via the Hyper-V Manager console which showed the dynamic memory demand of this RDS machine.  When the topmost listed instances of the SPLwow64.exe process were ‘ended’ the overall memory dropped equally dramatically.  This server which typically needed roughly 4 GB of running memory was underperforming when maxing out at 10 GB that were dynamically being made available to it.

The link below states in the first post that you can adjust the time this process takes to release its memory and links to a dead KB article.

http://forums.techarena.in/windows-x64-edition/816779.htm

That applicable control registry key is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\SplWOW64TimeOut

The second client’s issue led me to find the following thread and and in the last post the solution.  It’s a simple technique that can be applied if you want to turn off the use of system processes, SYSwow64.exe in this case, for a Terminal Server.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\SysProcs
Add a REG_DWORD called "splwow64.exe" set the value to 0

**Here’s another related conversation thread by an application developer found in the private Microsoft Forums

http://social.technet.microsoft.com/Forums/en-US/appvclients/thread/da0c76ea-5653-4439-a515-8246a2135cdc