Recently, I came across another Curious Case. Let’s start with some basic facts, shall we?
It’s a small, dedicated XenApp 6.5 farm, consisting of just 2 servers serving Published Desktops to around 50 concurrent users. They did qualify as Power Users for sure, with some of them having iexplore.exe’s running at 1 GB RAM.
They connect over some kind of MPLS network to a data center, with a dedicated print server next to the XenApp servers connecting users to a bucket load of printers over the MPLS link.
I’ve “rebuilt” the servers twice already. The first time was the result of some storage-related issues. I was able to revert back to a snapshot, or to do it correctly: setup clones based on a previous snapshot. Users were able to log on again.
After a number of weeks, the users reported that they could not log on again, stating “low on system resources”. We rolled back to a weekly snapshot, buying us some time to install 2 new servers from scratch, to rule out some cloning-related issues.
We moved all users to the new servers and all was good yet again. But that wouldn’t stay that way obviously, otherwise, this would be a rather lame article, wouldn’t it?
All of a sudden, users started logging on with Temporary Profiles. After ruling out the usual suspects, the issue started to become very complicated. A temporary profile was loaded even with a single user on that server, even after reboot. We got reports from different users, different timings… so we had a “random” error on our hands.
Since the servers were rebuilt from scratch we were now sure that reverting to snapshots would only temporarily fix the problem. At that point, we were sure it would return we just didn’t know when.
We decided to log an MS Support case and after a few case engineers gave it a shot, I was pointed in the right direction: registry size. To be more specific: the size of the default Local System registry profile. It turned out that the size of the DEFAULT file had grown to a staggering 1.8 – 2GB in size.
We all know that a Default User Profile is only used when a user doesn’t have a profile and a new one needs to be created. That’s still true of course, but .DEFAULT does not stand for the Default User Profile as I initially assumed (and wasn’t corrected by MS Support). Thanks to Aaron Parker (@stealthpuppy) who pointed this out to me, this is actually the Local System Profile. Still, a 2Gb Local System profile would consume way too much system resources obviously.
So now we did find our bad guy, and MS provided a procedure to restore a DEFAULT file:
- Boot one working machine into recovery console and copy the file c:\windows\system32\config\default to the root of the hard disk. Reboot normally
- Copy the default file from the working machine to the failing machine to the root of the hard disk
- Boot the failing machine using recovery console and copy the file c:\windows\system32\config\default to c:\windows\system32\config\default.old to have a backup.
- Copy the file c:\default to c:\windows\system32\config\default
- Reboot the machine
My tip: make a copy of the DEFAULT profile anyway after initial setup. That way you can always revert back to that one quite easily.
Next up: the search for the root cause. Using a tool provided by MS Support we were able to identify 2 registry keys that were clearly oversized.
HKEY_USERS.DEFAULT\Printers\DevModePerUser
HKEY_USERS.DEFAULT\Printers\DevModes2
First of all, you can just delete those keys or their content. Users will not have any impact, but the file size will remain the same, only with a lot of “white space”.
MS added some details to this: they did have some “rare cases” where print drivers attempt to write into the default profile, filling it up until the point where the registry is becoming too large causing registry pressure.
Guess what… we had such a “rare case” on our hands. Initially, I restored a lean and mean copy of the DEFAULT file, only to see it grow to 1.15GB in just 2 weeks. After another restore, I set a new XenApp policy: store printer preferences in user profile only (instead of “client device first, user profile second”). For a few weeks after that, file size stayed stable.
But this wasn’t the end of it, file size did grow, although slower than before. Starting with a clean DEFAULT file I was able to find the reason behind everything: a number of printer drivers. When a user connects to a printer, that connection adds 2 registry entries, see screenshots below.
On some printers, this happens only once. But for some others, this happens every time! In this specific case, every time a user logs on, a GPO connects the printers… and 2 registry entries are added. Again, this is driver-specific, meaning that some drivers do show this behavior while others don’t.
In this specific customer case, a majority of printers are using the same driver. And look what I found in the release notes of the latest update:
Fixed:
– When the driver is reinstalled it creates an additional instance of an existing registry key instead of just re-using it. Each time the driver is re-installed a new key is added to the registry.
Looks promising? Well, it has proven to be more than promising. After this update, this specific driver didn’t exhibit the weird behavior anymore. For those who want to know, it’s a RICOH SP4100N PCL5 driver.
We do still have 3 printers that use a driver that’s updating the default profile for some reason. There is no driver update available and I stay clear of so-called universal drivers. But they are on their way out anyway in a few weeks or months time, so at this point, the environment is stable, the file size doesn’t grow dramatically anymore and we just keep monitoring the file size and restore a clean version when it might become necessary.
Update: Hotfix available
Microsoft has released a hotfix for this issue: http://support.microsoft.com/kb/2871131.
Even though this post is 7 years old it is still an actual problem.
We have a customer with a Sharp printer (and Sharp drivers) and this article is exactly what we needed to point us in the right direction.
Without this info we would still be searching.
Thnx Carl and all others for posting you’re solutions.
Many thanks for article.
Great article, ticked so many of the boxes for the issue I’ve seen on three Windows 2008 R1 servers.
However, we also saw that one of the servers also had the HP registry key (thousands upon thousands of them!) stored in the shadow keys, so each user was picking these up and storing them in their profile as well. We were seeing NTUSER.DAT bloat for each user logging in. However due to how the registry works, even removing the offending keys from the registry for the users isn’t sufficient to recover that space, their NTUSER.DAT still stays at a large size (in our case, betwee 35 and 50MB). When you have 80 users logging into one server and take into consideration the previously bloated DEFAULT and HKLM hives, this soon maxed the registrysizelimit and stopped logons working.
I’ve deleted the HP keys from \.DEFAULT\SOFTWARE\Hewlett-Packard and \HKLM\SOFTWARE\Hewlett-Packard, as well as the shadow keys which in my case I found on both 32bit and 64bit locations. I then removed the HPs entirely from the servers as we use a Canon print solution where all devices are now Canon (or have difference files written for them by Canon), or we use Citrix UPD.
So far, things are remaining OK. I haven’t attempted the refresh of the HIVES for SOFTWARE and DEFAULT yet, as the system these servers host is going cloud in the next 12 months, and I unfortunately don’t have time to investigate it much further.
I wanted to let everyone know about a little utility I found to look for large reg keys. Unfortunately it will only give the size of the root folders underneath the Default key. When you run the scanner its pretty clear that it’s sitting on one folder processing the entries for way too long. Usually I just stop it and delete the entry. Then start again to make sure there’s no more large folders.
http://www.tliquest.net/software/rsp/
Had extremely good luck with the tweaking.com registry compressor. It took about 20 mins to go through a 1.5 GB registry file on Best compression. Almost no time at all to move and compress the file.
In my particular case these were the large folders:
HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
HKEY_USERS\.DEFAULT\Software\Sharp\CSR|mhsdc1
One last note: When I rebooted after running the registry compressor nothing could communicate with the server. The server seemed fine but I couldn’t RDP and the farm wasn’t connecting to either of my servers. I was panicked thinking I hosed both of my servers and was going to have to do a restore. After another reboot they came up fine and I haven’t had an issue since then. Maybe this will happen to you. Just ask yourself the question you always ask your users:
Have you tried rebooting? LOL
This is a great article I was having the same issue except it was different keys that were bloated. The keys I had problem with are:
HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
I was able to find these using the RU utility. Here is the process I used to clean up the registry:
1. Open Regedit
a. Delete the following Keys:
○ HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
○ HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
b. Add the following Keys:
○ HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
○ HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
2. Reboot
3. Open regedit
4. Browse to HKEY_USERS\.DEFAULT
5. Right click on .DEFAULT Select Export
6. Export the .DEFAULT hive as a “Registry Hive” file with a unique name. %windir%\system32\config\compressedDEFAULT
7. You can use Windows Explorer verify the old and new sizes of the registry hives. (DEFAULT = ~600 MB, compressedDefault = ~3MB)
8. Shutdown Server
9. Start the server by using Windows Server 2008 R2 SP1 media.
10. Select Repair your computer.
11. Select Command Prompt.
12. At the command prompt, run the following commands to Rename the hives so that you will boot with the compressed hive.
a. %windir%\system32\config\ren DEFAULT DEFAULT.old
b. %windir%\system32\config\ren compressedDEFAULT DEFAULT
13. Reboot Server
14. Open Regedit verify the following Keys are empty
• HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
• HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
Installed hotfix http://support.microsoft.com/kb/2871131
GReat article – I believe this is exactly what we are experiencing. Temp profiles and also “Profile Service failed the Logon” errors – we exported our HKEY_USER\Default hive and it is 11GB…..
Looks to be a problem with print driver. Hopefully these tools and steps will fix problem!
Hi, I´m trying to load the bloated hive, I already booted from a WinPe, opened regedit but when trying to load the bloated hive under HKLM i get “cannot load x:\windows\system32\config\software: The process cannot access the file because it is being used by another process”
What am I doing wrong?.
Great article, helped me through the same issue very quickly.
Thanks for putting the time into writing it.
Hi,
i’m experiencing the same issue on a 2008R2 RD broker FARM. I’ve installed the HOTFIX and compressed the HIVE successfully on the 1st Terminal Server. The other 2 servers didn’t behave as expected. The other 2 servers blue screened after hive compression. I had to revert the machines to a earlier snapshot (cloned/sysprepped from the 1 server) servers. After the installation of the HOTFIX i’ve noticed that the HIVE still is filling up with entries of the Dymo Labelwriter driver… Does anyone else experienced a same issue on a xenapp or terminal server farm? I think that the only woraround possible is eliminate the bad printer and driver (not Xenapp/terminal server aware drivers), use VDISKS and/or create some kind of script who cleanes up the bloated registry hive.
Regards… Naures
Hi
Had exactly this problem with my 2008R2 RDS server, the Default file was 1,5Gb big, Found an easy solution to the problem 🙂 I used the Tweaking.com – Registry Compressor and ran it with “Best” compression it compressed my 1,5GB file down to 256kb and now everthing is working normal again.
Here´s a link to registry compressor
http://www.tweaking.com/content/page/tweaking_com_registry_compressor.html
/Martin
Hi Martin/All
Please note that we used the 3rd party tool mentioned from Tweaking.com and it worked great completely compressing the file. However the next day we noticed 100% CPU on every server we ran the tool on. On further investigation, the winlogon.exe process was using about 3-5% CPU for each user logged onto the server. We found that the winlogon.exe process was ‘looping’ since it was missing some critical reg keys in the HKEY_USERS\.DEFAULT hive. The tool pretty much deletes everything even required critical keys! We ended up having to go and use the official Microsoft method to resolve the issue. Or, goto a working server and export the reg to a file, and import it after running the 3rd party tool. CPU dropped from 100% to 0 instantly after importing the reg keys.
Boot one working machine into recovery console and copy the file c:\windows\system32\config\default to the root of the hard disk. Reboot normally
Copy the default file from the working machine to the failing machine to the root of the hard disk
Boot the failing machine using recovery console and copy the file c:\windows\system32\config\default to c:\windows\system32\config\default.old to have a backup.
Copy the file c:\default to c:\windows\system32\config\default
Reboot the machine
Check out the steps here.
I am experiencing the same issue with TEMP profiles in a Xenapp 6 environment.
The .DEFAULT key is 1.96GB in size.
http://social.technet.microsoft.com/Forums/windowsserver/en-US/971a6470-7e5c-4bd6-85bb-82228aa7c55c/rds-insufficient-system-resources-exist-to-complete-the-requested-service-for-users-ntuserdat
http://support.microsoft.com/kb/2498915 — This is the MS article outlining how to compress the registry hive. If you don’t have access to WINPE, you can boot the OS up with the Windows Installation ISO. Once the Install screen comes up, press ALT + F10 and a command prompt will open up. You can then follow microsoft’s instructions to compress the hive.
This is the process I took to compress the hive, and it went from 1.5GB to less than 10MB! Hope this helps.
Thanks for the great article, helps us a lot!
Hi There – To address the white space your speaking about due to the registry hive already being expanded by the bloated keys, The following process outlined in the microsoft KB 2498915 can be followed: http://support.microsoft.com/kb/2498915
It works great, did it on a bunch of XA Server’s where I had the same print driver bloat issues…
Thanks,
Shane
Interesting that you had MS support-case. Had a similar customer where the installer for any (MSI) application would simply stall for a long time. I extracted about 300 MB of registry information that was forced to be checked by the Windows Installer service and gave very much the same experience as you described for the users.
Interesting that it was a Ricoh-printer driver, must say….
That’s not the default profile, C:\Windows\System32\config contains the profile for SYSTEM. The profile that is copied for a user with no pre-existing profile is C:\Users\Default
Aaron,
I stand corrected.
The key HKEY_USERS\.DEFAULT is stored in the DEFAULT file in c:\windows\system32\config and is the profile for the Local Systemaccount like you stated.
I’ll change the title of the article accordingly.
Funny thing though, MS support called it “the default profile” themselves…go figure.
I hope this makes things a bit more clear?
It’s a worry when Microsoft don’t get it right, but it’s a common misconception. Citrix don’t get it right either.
Aaron, after your initial comment I asked the senior(!) support engineer the exact question… and he gave the wrong answer…
Great article – I had the same issue with an HP driver.
To prevent such an issue in the future the following perfmon counter should be monitored:
System > % Registry Quota In Use
Tracks registry size, to help discover bloat a new sysinternals tool regiatry usage (RU) http://technet.microsoft.com/en-us/sysinternals/dn194428 can be used to find bloat. At the time of my issue had to start exporting keys to determine suspect. When exported size is doubled cause registry compresses.
On phone – will comment more on this, there is another process to delete keys and re compress the hive shrinking it back down.
Shane
Agreed on the ‘great article’ note.
Quite happy to, via your response, find the Ru utility. DUREG.exe was recommended by our Microsoft support (from the windows 2000 resource kit…) and it failed on half of my test runs.
Shane: I would love to hear about another process to delete keys and re-compress a registry hive. Hope you add to your post.