Carl Webster Accessibility Statement

Carl Webster is committed to facilitating the accessibility and usability of its website, carlwebster.com, for everyone. Carl Webster aims to comply with all applicable standards, including the World Wide Web Consortium’s Web Content Accessibility Guidelines 2.0 up to Level AA (WCAG 2.0 AA). Carl Webster is proud of the efforts that we have completed and that are in-progress to ensure that our website is accessible to everyone.

If you experience any difficulty in accessing any part of this website, please feel free to email us at info@carlwebster.com and we will work with you to provide the information or service you seek through an alternate communication method that is accessible for you consistent with applicable law (for example, through telephone support).

  • Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load

    March 2, 2016

    XenApp 6.0, XenApp 6.5

    On a recent project, the customer experienced an issue where updating the Citrix Provisioning Services (PVS) image to Internet Explorer (IE) 11 caused a major outage.

    If any users caused two Microsoft Office applications to run simultaneously, the CPU would spike between 80% to 100%, and the server would become unresponsive. The only recourse was to use the hypervisor to either Force Shutdown or Force Reset the server.

    The customer’s previous PVS image used the same version of Microsoft Office and other core applications. The only thing that changed was an upgrade from IE10 to IE11. After reverting users to different servers for the Office applications, troubleshooting the root cause began.

    To make a very long story short, it was NOT:

    • An IE11 issue
    • A PVS issue
    • A XenApp issue
    • An Office application issue
    • Or a user issue

    It turns out there is a known issue on Server 2008 and Server 2008 R2 with the NTFS IO system being unable to handle the stress of large amounts of file IO. Both Citrix and Microsoft were involved in the troubleshooting and reading of dumps. Microsoft sent a hotfix that the support representative said should be installed on every 2008 R2 server whether they experience this issue. This has been a known issue since Server 2008.

    Indeed the hotfix is readily available if this has been a known issue for so long. No, it is not. Indeed this major bug was fixed in a product update or service pack for Server 2008 or Server 2008 R2? Not it was not. The hotfix is only available by opening a Microsoft support ticket. The good news is Microsoft fixed the bug in Server 2012.

    The bottom line was the customer doubled the number of servers being streamed by PVS and upgraded to IE11, and the new users had large mailboxes and huge Excel files. The NTFS file system could not keep up with the extra demand, and the XenApp 6.5 servers would hard lock when a large Excel file and a large mailbox were opened on the same server.

    If you experience this issue, look for A computer that is running Windows Server 2008 R2 SP1 or Windows 7 SP1 freezes under a heavy file I/O load. You know Microsoft, this hotfix is only available from them only if you open a support ticket. I can’t give it to you. 🙂

    Thanks

    Webster







    About Carl Webster

    Webster is a Sr. Solutions Architect for Choice Solutions, LLC and specializes in Citrix, Active Directory and Technical Documentation. Webster has been working with Citrix products for many years starting with Multi-User OS/2 in 1990.

    View all posts by Carl Webster

    9 Responses to “Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load”

    1. Lee Says:

      Hi Carl,

      we have a similar issue here, almost identical to what you have seen.
      Can you give any idea where I can find information on the known issues with heavy IO as it may help point me in the direction of how this update can actually be applied and if it is valid?
      as obviously we wouldn’t want to downgrade our ntfs.sys file on a whim, if we could work out how to do it at all…

      Reply

      • Carl Webster Says:

        I was not working the case and the only email I saw from Microsoft and Citrix referenced the Microsoft hotfix and to also make sure all antivirus exclusions were configured. We did find their traditional antivirus software was causing a severe performance issues with a specific application (well over 14000 actions while launching the application). Disabling the antivirus services cut the launch time of the application from 45 seconds to 15 seconds. Obviously, making sure exclusions are properly configured are important. The customer has decided to remove their traditional antivirus software from all servers and are moving to https://www.cylance.com/ . Cylance also requires precautions with “Citrix” servers and specific applications but has a very small RAM and CPU footprint. Watching it prevent ransomware attacks in real-time is amazing.

        Thanks

        Webster

        Reply

    2. Jonathan Pitre Says:

      Here is the lastest version of ntfs.sys for 2008 R2 and yes fixes are cumulative.
      https://support.microsoft.com/en-us/kb/3121255

      No need to call MS to get a deprecated hotfix.

      In fact there is more than 800 hotfixes for 2008 R2. I strongly suggest you just use KUC http://www.windows-update-checker.com and just install all of them into your golden images. The recommended hotfixes list by Citrix is missing just a few hundreds…

      Reply

      • Carl Webster Says:

        In a large enterprise with very strict change control, they used the update Microsoft told them to use. Installing Citrix or Microsoft updates into images where over 1000 applications are used by people around the world is not an easy task. They are still testing updated apps so they can finish their move off of PS4.5 and Server 2003.

        Webster

        Reply

    3. Jeremy Saunders Says:

      Hi Webster,

      Nice write-up.

      This is just an updated ntfs.sys file. I wonder if the security update from kb2840149 would have also solved the issue given that it’s a newer version? Possibly no need to contact Microsoft.

      Cheers,
      Jeremy

      Reply

      • Carl Webster Says:

        The ntfs.sys file in the hotfix was older than what was in the PVS image having the issue.

        Webster

        Reply

        • Jeremy Saunders Says:

          That’s weird. Then logic tells me it shouldn’t have applied. Hotfixes only update older files. So the update rolled back the version of the nfts.sys file, or did it do something else?

          Cheers,
          Jeremy

          Reply

          • Carl Webster Says:

            I was working on GPO issues so I was not here when they applied the hotfix. Both Citrix and Microsoft were on the call with the customer so I was not directly involved. I was just asked to write it up.

            Webster

            Reply

        • Jeremy Saunders Says:

          No problems. I’m just curious. I got hold of the hotfix and tried to run it on a test machine that already had a newer version of ntfs.sys. WUSA said that it wasn’t applicable. Sorry, I’m just intrigued how this hotfix applied.

          Cheers,
          Jeremy

          Reply

    Leave a Reply to Lee