• The Curious Case of the Bloated Local System (not Default) Profile – Hotfixed?

    Recently, I came across another Curious Case. Let’s start with some basic facts, shall we?

    It’s a small, dedicated XenApp 6.5 farm, consisting of just 2 servers serving Published Desktops to around 50 concurrent users. They did qualifiy as Power Users for sure, with some of them having iexplore.exe’s running at 1 GB RAM.

    They connect over some kind of MPLS network to a datacenter, with a dedicated print server next to the XenApp servers connecting users to a bucket load of printers over the MPLS link.

    I’ve “rebuilt” the servers twice already. The first time was the result of some storage related issues. I was able to revert back to a snapshot, or to do it correctly: setup clones based on a previous snapshot. Users were able to logon again.

    After a number of weeks, the users reported that they could not logon again, stating “low on system resources”. We rolled back to a weekly snapshot, buying us some time to install 2 new servers from scratch, to rule out some cloning related issues.

    We moved all users to the new servers and all was good yet again. But that wouldn’t stay that way obviously, otherwise this would be a rather lame article, wouldn’t it?

    All of a sudden, users started logging on with Temporary Profiles. After ruling out the usual suspects, the issue started to become very complicated. A temporary profile was loaded even with a single user on that server, even after reboot. We got reports from different users, different timings… so we had a “random” error on our hands.

    Since the servers were rebuilt from scratch we were now sure that reverting to snapshots would only temporary fix the problem. At that point, we were sure it would return we just didn’t know when.

    We decided to log a MS Support case and after a few case engineers gave it a shot, I was pointed in the right direction: registry size. To be more specific: the size of the default Local System registry profile. It turned out that the size of the DEFAULT file had grown to a staggering 1.8 – 2GB in size.

    Filesize

    We all know that a Default User Profile is only used when a user doesn’t have a profile and a new one needs to be created. That’s still true of course, but .DEFAULT does not stand for the Default User Profile as I initially assumed (and wasn’t corrected by MS Support). Thanks to Aaron Parker (@stealthpuppy) who pointed this out to me, this is actually the Local System Profile. Still, a 2Gb Local System profile would consume way too much system resources obviously.

    So now we did find our bad guy, and MS provided a procedure to restore a DEFAULT file:

    1. Boot one working machine into recovery console and copy the file c:\windows\system32\config\default to the root of the hard disk. Reboot normally
    2. Copy the default file from the working machine to the failing machine to the root of the hard disk
    3. Boot the failing machine using recovery console and copy the file c:\windows\system32\config\default to c:\windows\system32\config\default.old to have a backup.
    4. Copy the file c:\default to c:\windows\system32\config\default
    5. Reboot the machine

    My tip: make a copy of the DEFAULT profile anyway after initial setup. That way you can always revert back to that one quite easily.

    Next up: the search for the root case. Using a tool provided by MS Support we were able to identify 2 registry keys that were clearly oversized.

    HKEY_USERS\.DEFAULT\Printers\DevModePerUser

    HKEY_USERS\.DEFAULT\Printers\DevModes2

    First of all, you can just delete those keys or their content. Users will not have any impact, but file size will remain the same, only with a lot of “white space”.

    MS added some details to this: they did have some “rare cases” where print drivers attempt to write into the default profile, filling it up until the point where the registry is becoming too large causing registry pressure.

    Guess what… we had such a “rare case” on our hands. Initially I restored a lean and mean copy of the DEFAULT file, only to see it grow to 1.15GB in just 2 weeks. After another restore, I set a new XenApp policy: store printer preferences in user profile only (instead of “client device first, user profile second”). For a few weeks after that, file size stayed stable.

    But this wasn’t the end of it, file size did grow, although slower than before. Starting with a clean DEFAULT file I was able to find the reason behind everything: a number of printer drivers. When a user connects to a printer, that connection adds 2 registry entries, see screenshots below.

    DevModes2

    DevModePerUser

    On some printers, this happens only once. But for some others, this happens every time! In this specific case, every time a user logs on, a GPO connects the printers… and 2 registry entries are added. Again, this is driver specific, meaning that some drivers do show this behavior while others don’t.

    In this specific customer case, a majority of printers are using the same driver. And look what I found in the release notes of the latest update:

    Fixed:

    – When the driver is reinstalled it creates an additional instance of an existing registry key instead of just re-using it. Each time the driver is re-installed a new key is added to the registry.

    Looks promising? Well it has proven to be more then promising. After this update this specific driver didn’t exhibit the weird behavior anymore. For those who want to know, it’s a RICOH SP4100N PCL5 driver.

    We do still have 3 printers that use a driver that’s updating the default profile for some reason. There is no driver update available and I stay clear of so-called universal drivers. But they are on their way out anyway in a few weeks or months time, so at this point, the environment is stable, file size doesn’t grow dramatically anymore and we just keep monitoring the file size and restore a clean version when it might become necessary.

    *Update: Hotfix available*

    Microsoft has released a hotfix for this issue: http://support.microsoft.com/kb/2871131.

    , , , ,





    About Bart Jacobs

    Bart Jacobs is a Senior System Engineer/Consultant based in Belgium. He started his career back in 1998. One of the first projects he worked on in those days was Citrix Metaframe 1.8 on Microsoft Windows NT 4 Terminal Server codename "Hydra". Over the years, Citrix technology has always been a major theme in his professional career, resulting in becoming a true technical expert in the matter. In the last few years, he has also become an expert in virtualization technology, with a special interest in a real challenger in this business: Citrix XenServer. Bart has founded his own company BJ IT back in 2007 and is mainly working as a (Citrix) consultant now. In 2019, Bart received his Citrix CTA award.

    View all posts by Bart Jacobs

    22 Responses to “The Curious Case of the Bloated Local System (not Default) Profile – Hotfixed?”

    1. Gultekin Says:

      Many thanks for article.

      Reply

    2. Alex Hawkins Says:

      Great article, ticked so many of the boxes for the issue I’ve seen on three Windows 2008 R1 servers.

      However, we also saw that one of the servers also had the HP registry key (thousands upon thousands of them!) stored in the shadow keys, so each user was picking these up and storing them in their profile as well. We were seeing NTUSER.DAT bloat for each user logging in. However due to how the registry works, even removing the offending keys from the registry for the users isn’t sufficient to recover that space, their NTUSER.DAT still stays at a large size (in our case, betwee 35 and 50MB). When you have 80 users logging into one server and take into consideration the previously bloated DEFAULT and HKLM hives, this soon maxed the registrysizelimit and stopped logons working.

      I’ve deleted the HP keys from \.DEFAULT\SOFTWARE\Hewlett-Packard and \HKLM\SOFTWARE\Hewlett-Packard, as well as the shadow keys which in my case I found on both 32bit and 64bit locations. I then removed the HPs entirely from the servers as we use a Canon print solution where all devices are now Canon (or have difference files written for them by Canon), or we use Citrix UPD.

      So far, things are remaining OK. I haven’t attempted the refresh of the HIVES for SOFTWARE and DEFAULT yet, as the system these servers host is going cloud in the next 12 months, and I unfortunately don’t have time to investigate it much further.

      Reply

    3. Michael Fahey Says:

      I wanted to let everyone know about a little utility I found to look for large reg keys. Unfortunately it will only give the size of the root folders underneath the Default key. When you run the scanner its pretty clear that it’s sitting on one folder processing the entries for way too long. Usually I just stop it and delete the entry. Then start again to make sure there’s no more large folders.

      http://www.tliquest.net/software/rsp/

      Had extremely good luck with the tweaking.com registry compressor. It took about 20 mins to go through a 1.5 GB registry file on Best compression. Almost no time at all to move and compress the file.

      In my particular case these were the large folders:
      HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
      HKEY_USERS\.DEFAULT\Software\Sharp\CSR|mhsdc1

      Reply

      • Michael Fahey Says:

        One last note: When I rebooted after running the registry compressor nothing could communicate with the server. The server seemed fine but I couldn’t RDP and the farm wasn’t connecting to either of my servers. I was panicked thinking I hosed both of my servers and was going to have to do a restore. After another reboot they came up fine and I haven’t had an issue since then. Maybe this will happen to you. Just ask yourself the question you always ask your users:

        Have you tried rebooting? LOL

        Reply

    4. J Dunlap Says:

      This is a great article I was having the same issue except it was different keys that were bloated. The keys I had problem with are:
      HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
      HKEY_USERS\.DEFAULT\Software\SSPrint\spep6

      I was able to find these using the RU utility. Here is the process I used to clean up the registry:

      1. Open Regedit
      a. Delete the following Keys:
      ○ HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
      ○ HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
      b. Add the following Keys:
      ○ HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
      ○ HKEY_USERS\.DEFAULT\Software\SSPrint\spep6
      2. Reboot
      3. Open regedit
      4. Browse to   HKEY_USERS\.DEFAULT
      5. Right click on .DEFAULT Select Export
      6. Export the .DEFAULT hive as a “Registry Hive” file with a unique name.  %windir%\system32\config\compressedDEFAULT
      7. You can use Windows Explorer verify the old and new sizes of the registry hives. (DEFAULT = ~600 MB, compressedDefault = ~3MB)
      8. Shutdown Server
      9. Start the server by using Windows Server 2008 R2 SP1 media.
      10. Select Repair your computer.
      11. Select Command Prompt.
      12. At the command prompt, run the following commands to Rename the hives so that you will boot with the compressed hive.
      a. %windir%\system32\config\ren DEFAULT DEFAULT.old
      b. %windir%\system32\config\ren compressedDEFAULT DEFAULT
      13. Reboot Server
      14. Open Regedit verify the following Keys are empty
      • HKEY_USERS\.DEFAULT\Software\Hewlett-Packard
      • HKEY_USERS\.DEFAULT\Software\SSPrint\spep6

      Installed hotfix http://support.microsoft.com/kb/2871131

      Reply

    5. Tumac Says:

      GReat article – I believe this is exactly what we are experiencing. Temp profiles and also “Profile Service failed the Logon” errors – we exported our HKEY_USER\Default hive and it is 11GB…..

      Looks to be a problem with print driver. Hopefully these tools and steps will fix problem!

      Reply

    6. Sergio Says:

      Hi, I´m trying to load the bloated hive, I already booted from a WinPe, opened regedit but when trying to load the bloated hive under HKLM i get “cannot load x:\windows\system32\config\software: The process cannot access the file because it is being used by another process”

      What am I doing wrong?.

      Reply

    7. Arron Says:

      Great article, helped me through the same issue very quickly.

      Thanks for putting the time into writing it.

      Reply

    8. Naures Says:

      Hi,

      i’m experiencing the same issue on a 2008R2 RD broker FARM. I’ve installed the HOTFIX and compressed the HIVE successfully on the 1st Terminal Server. The other 2 servers didn’t behave as expected. The other 2 servers blue screened after hive compression. I had to revert the machines to a earlier snapshot (cloned/sysprepped from the 1 server) servers. After the installation of the HOTFIX i’ve noticed that the HIVE still is filling up with entries of the Dymo Labelwriter driver… Does anyone else experienced a same issue on a xenapp or terminal server farm? I think that the only woraround possible is eliminate the bad printer and driver (not Xenapp/terminal server aware drivers), use VDISKS and/or create some kind of script who cleanes up the bloated registry hive.

      Regards… Naures

      Reply

    9. Martin Björgvik Says:

      Hi
      Had exactly this problem with my 2008R2 RDS server, the Default file was 1,5Gb big, Found an easy solution to the problem 🙂 I used the Tweaking.com – Registry Compressor and ran it with “Best” compression it compressed my 1,5GB file down to 256kb and now everthing is working normal again.

      Here´s a link to registry compressor
      http://www.tweaking.com/content/page/tweaking_com_registry_compressor.html

      /Martin

      Reply

      • Jake Says:

        Hi Martin/All

        Please note that we used the 3rd party tool mentioned from Tweaking.com and it worked great completely compressing the file. However the next day we noticed 100% CPU on every server we ran the tool on. On further investigation, the winlogon.exe process was using about 3-5% CPU for each user logged onto the server. We found that the winlogon.exe process was ‘looping’ since it was missing some critical reg keys in the HKEY_USERS\.DEFAULT hive. The tool pretty much deletes everything even required critical keys! We ended up having to go and use the official Microsoft method to resolve the issue. Or, goto a working server and export the reg to a file, and import it after running the 3rd party tool. CPU dropped from 100% to 0 instantly after importing the reg keys.

        Boot one working machine into recovery console and copy the file c:\windows\system32\config\default to the root of the hard disk. Reboot normally
        Copy the default file from the working machine to the failing machine to the root of the hard disk
        Boot the failing machine using recovery console and copy the file c:\windows\system32\config\default to c:\windows\system32\config\default.old to have a backup.
        Copy the file c:\default to c:\windows\system32\config\default
        Reboot the machine

        Reply

    10. Gerard Says:

      Check out the steps here.

      I am experiencing the same issue with TEMP profiles in a Xenapp 6 environment.

      The .DEFAULT key is 1.96GB in size.

      http://social.technet.microsoft.com/Forums/windowsserver/en-US/971a6470-7e5c-4bd6-85bb-82228aa7c55c/rds-insufficient-system-resources-exist-to-complete-the-requested-service-for-users-ntuserdat

      Reply

    11. Shane Kleinert Says:

      http://support.microsoft.com/kb/2498915 — This is the MS article outlining how to compress the registry hive. If you don’t have access to WINPE, you can boot the OS up with the Windows Installation ISO. Once the Install screen comes up, press ALT + F10 and a command prompt will open up. You can then follow microsoft’s instructions to compress the hive.

      This is the process I took to compress the hive, and it went from 1.5GB to less than 10MB! Hope this helps.

      Reply

    12. Arno-R Says:

      Thanks for the great article, helps us a lot!

      Reply

    13. Shane Kleinert Says:

      Hi There – To address the white space your speaking about due to the registry hive already being expanded by the bloated keys, The following process outlined in the microsoft KB 2498915 can be followed: http://support.microsoft.com/kb/2498915

      It works great, did it on a bunch of XA Server’s where I had the same print driver bloat issues…

      Thanks,

      Shane

      Reply

    14. Nicke Källén Says:

      Interesting that you had MS support-case. Had a similar customer where the installer for any (MSI) application would simply stall for a long time. I extracted about 300 MB of registry information that was forced to be checked by the Windows Installer service and gave very much the same experience as you described for the users.

      Interesting that it was a Ricoh-printer driver, must say….

      Reply

    15. Aaron Says:

      That’s not the default profile, C:\Windows\System32\config contains the profile for SYSTEM. The profile that is copied for a user with no pre-existing profile is C:\Users\Default

      Reply

      • Bart Jacobs Says:

        Aaron,

        I stand corrected.

        The key HKEY_USERS\.DEFAULT is stored in the DEFAULT file in c:\windows\system32\config and is the profile for the Local Systemaccount like you stated.
        I’ll change the title of the article accordingly.

        Funny thing though, MS support called it “the default profile” themselves…go figure.

        I hope this makes things a bit more clear?

        Reply

    16. Shane kleinert Says:

      Great article – I had the same issue with an HP driver.

      To prevent such an issue in the future the following perfmon counter should be monitored:

      System > % Registry Quota In Use

      Tracks registry size, to help discover bloat a new sysinternals tool regiatry usage (RU) http://technet.microsoft.com/en-us/sysinternals/dn194428 can be used to find bloat. At the time of my issue had to start exporting keys to determine suspect. When exported size is doubled cause registry compresses.

      On phone – will comment more on this, there is another process to delete keys and re compress the hive shrinking it back down.

      Shane

      Reply

      • R. Schultz Says:

        Agreed on the ‘great article’ note.

        Quite happy to, via your response, find the Ru utility. DUREG.exe was recommended by our Microsoft support (from the windows 2000 resource kit…) and it failed on half of my test runs.

        Shane: I would love to hear about another process to delete keys and re-compress a registry hive. Hope you add to your post.

        Reply

    Leave a Reply