Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load

On a recent project, the customer experienced an issue where after updating the Citrix Provisioning Services (PVS) image to Internet Explorer (IE) 11 caused a major outage.

What would happen is that is any user or users caused two Microsoft Office applications to run at the same time, the CPU would spike to between 80% to 100% and the server would become unresponsive. The only recourse was to use the hypervisor to either Force Shutdown or Force Reset the server.

The customer’s previous PVS image used the exact same version of Microsoft Office and other core applications. The only thing that changed was an upgrade from IE10 to IE11. After reverting users to other servers for the Office applications, troubleshooting the root cause began.

To make a very long story short, it was NOT:

  • An IE11 issue
  • A PVS issue
  • A XenApp issue
  • An Office application issue
  • Or a user issue

Turns out there is a known issue on Server 2008 and Server 2008 R2 with the NTFS IO system not being able to handle the stress of large amounts of file IO. Both Citrix and Microsoft were involved in the troubleshooting and reading of dumps. Microsoft sent a hotfix that the support representative said should be installed on every 2008 R2 server whether they experience this issue or not. This has been a known issue since Server 2008.

If this has been a known issue for so long, surely the hotfix is readily available? No it is not. Surely this major bug was fixed in a product update or service pack for Server 2008 or Server 2008 R2? Not it was not. The hotfix is only available by opening a Microsoft support ticket. Good news is Microsoft fixed the bug in Server 2012.

Bottom line was the customer doubled the number of servers being streamed by PVS and upgraded to IE11 and the new users had large mailboxes and extremely large Excel files. The NTFS file system could not keep up with the extra demand and the XenApp 6.5 servers would hard lock when a large Excel file and a large mailbox were opened on the same server.

If you experience this issue, look for A computer that is running Windows Server 2008 R2 SP1 or Windows 7 SP1 freezes under a heavy file I/O load . You know Microsoft, this hotfix is only available from them only if you open a support ticket. I can’t give it to you. 🙂

Thanks

 

Webster

About Carl Webster

Webster is a Sr. Solutions Architect for Choice Solutions, LLC and specializes in Citrix, Active Directory and Technical Documentation. Webster has been working with Citrix products for many years starting with Multi-User OS/2 in 1990.

View all posts by Carl Webster

9 Responses to “Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load”

  1. Lee Says:

    Hi Carl,

    we have a similar issue here, almost identical to what you have seen.
    Can you give any idea where I can find information on the known issues with heavy IO as it may help point me in the direction of how this update can actually be applied and if it is valid?
    as obviously we wouldn’t want to downgrade our ntfs.sys file on a whim, if we could work out how to do it at all…

    Reply

    • Carl Webster Says:

      I was not working the case and the only email I saw from Microsoft and Citrix referenced the Microsoft hotfix and to also make sure all antivirus exclusions were configured. We did find their traditional antivirus software was causing a severe performance issues with a specific application (well over 14000 actions while launching the application). Disabling the antivirus services cut the launch time of the application from 45 seconds to 15 seconds. Obviously, making sure exclusions are properly configured are important. The customer has decided to remove their traditional antivirus software from all servers and are moving to https://www.cylance.com/ . Cylance also requires precautions with “Citrix” servers and specific applications but has a very small RAM and CPU footprint. Watching it prevent ransomware attacks in real-time is amazing.

      Thanks

      Webster

      Reply

  2. Jonathan Pitre Says:

    Here is the lastest version of ntfs.sys for 2008 R2 and yes fixes are cumulative.
    https://support.microsoft.com/en-us/kb/3121255

    No need to call MS to get a deprecated hotfix.

    In fact there is more than 800 hotfixes for 2008 R2. I strongly suggest you just use KUC http://www.windows-update-checker.com and just install all of them into your golden images. The recommended hotfixes list by Citrix is missing just a few hundreds…

    Reply

    • Carl Webster Says:

      In a large enterprise with very strict change control, they used the update Microsoft told them to use. Installing Citrix or Microsoft updates into images where over 1000 applications are used by people around the world is not an easy task. They are still testing updated apps so they can finish their move off of PS4.5 and Server 2003.

      Webster

      Reply

  3. Jeremy Saunders Says:

    Hi Webster,

    Nice write-up.

    This is just an updated ntfs.sys file. I wonder if the security update from kb2840149 would have also solved the issue given that it’s a newer version? Possibly no need to contact Microsoft.

    Cheers,
    Jeremy

    Reply

    • Carl Webster Says:

      The ntfs.sys file in the hotfix was older than what was in the PVS image having the issue.

      Webster

      Reply

      • Jeremy Saunders Says:

        That’s weird. Then logic tells me it shouldn’t have applied. Hotfixes only update older files. So the update rolled back the version of the nfts.sys file, or did it do something else?

        Cheers,
        Jeremy

      • Carl Webster Says:

        I was working on GPO issues so I was not here when they applied the hotfix. Both Citrix and Microsoft were on the call with the customer so I was not directly involved. I was just asked to write it up.

        Webster

      • Jeremy Saunders Says:

        No problems. I’m just curious. I got hold of the hotfix and tried to run it on a test machine that already had a newer version of ntfs.sys. WUSA said that it wasn’t applicable. Sorry, I’m just intrigued how this hotfix applied.

        Cheers,
        Jeremy

Leave a Reply