-
Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load
March 2, 2016
On a recent project, the customer experienced an issue where updating the Citrix Provisioning Services (PVS) image to Internet Explorer (IE) 11 caused a major outage.
If any users caused two Microsoft Office applications to run simultaneously, the CPU would spike between 80% to 100%, and the server would become unresponsive. The only recourse was to use the hypervisor to either Force Shutdown or Force Reset the server.
The customer’s previous PVS image used the same version of Microsoft Office and other core applications. The only thing that changed was an upgrade from IE10 to IE11. After reverting users to different servers for the Office applications, troubleshooting the root cause began.
To make a very long story short, it was NOT:
- An IE11 issue
- A PVS issue
- A XenApp issue
- An Office application issue
- Or a user issue
It turns out there is a known issue on Server 2008 and Server 2008 R2 with the NTFS IO system being unable to handle the stress of large amounts of file IO. Both Citrix and Microsoft were involved in the troubleshooting and reading of dumps. Microsoft sent a hotfix that the support representative said should be installed on every 2008 R2 server whether they experience this issue. This has been a known issue since Server 2008.
Indeed the hotfix is readily available if this has been a known issue for so long. No, it is not. Indeed this major bug was fixed in a product update or service pack for Server 2008 or Server 2008 R2? Not it was not. The hotfix is only available by opening a Microsoft support ticket. The good news is Microsoft fixed the bug in Server 2012.
The bottom line was the customer doubled the number of servers being streamed by PVS and upgraded to IE11, and the new users had large mailboxes and huge Excel files. The NTFS file system could not keep up with the extra demand, and the XenApp 6.5 servers would hard lock when a large Excel file and a large mailbox were opened on the same server.
If you experience this issue, look for A computer that is running Windows Server 2008 R2 SP1 or Windows 7 SP1 freezes under a heavy file I/O load. You know Microsoft, this hotfix is only available from them only if you open a support ticket. I can’t give it to you. 🙂
Thanks
Webster
Subscribe
Subscribe to our e-mail newsletter to receive updates.
9 Responses to “Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load”
March 15, 2016 at 11:44 am
Hi Carl,
we have a similar issue here, almost identical to what you have seen.
Can you give any idea where I can find information on the known issues with heavy IO as it may help point me in the direction of how this update can actually be applied and if it is valid?
as obviously we wouldn’t want to downgrade our ntfs.sys file on a whim, if we could work out how to do it at all…
March 15, 2016 at 11:54 am
I was not working the case and the only email I saw from Microsoft and Citrix referenced the Microsoft hotfix and to also make sure all antivirus exclusions were configured. We did find their traditional antivirus software was causing a severe performance issues with a specific application (well over 14000 actions while launching the application). Disabling the antivirus services cut the launch time of the application from 45 seconds to 15 seconds. Obviously, making sure exclusions are properly configured are important. The customer has decided to remove their traditional antivirus software from all servers and are moving to https://www.cylance.com/ . Cylance also requires precautions with “Citrix” servers and specific applications but has a very small RAM and CPU footprint. Watching it prevent ransomware attacks in real-time is amazing.
Thanks
Webster
March 2, 2016 at 11:09 pm
Here is the lastest version of ntfs.sys for 2008 R2 and yes fixes are cumulative.
https://support.microsoft.com/en-us/kb/3121255
No need to call MS to get a deprecated hotfix.
In fact there is more than 800 hotfixes for 2008 R2. I strongly suggest you just use KUC http://www.windows-update-checker.com and just install all of them into your golden images. The recommended hotfixes list by Citrix is missing just a few hundreds…
March 3, 2016 at 6:45 am
In a large enterprise with very strict change control, they used the update Microsoft told them to use. Installing Citrix or Microsoft updates into images where over 1000 applications are used by people around the world is not an easy task. They are still testing updated apps so they can finish their move off of PS4.5 and Server 2003.
Webster
March 2, 2016 at 11:07 am
Hi Webster,
Nice write-up.
This is just an updated ntfs.sys file. I wonder if the security update from kb2840149 would have also solved the issue given that it’s a newer version? Possibly no need to contact Microsoft.
Cheers,
Jeremy
March 2, 2016 at 11:14 am
The ntfs.sys file in the hotfix was older than what was in the PVS image having the issue.
Webster
March 2, 2016 at 3:03 pm
That’s weird. Then logic tells me it shouldn’t have applied. Hotfixes only update older files. So the update rolled back the version of the nfts.sys file, or did it do something else?
Cheers,
Jeremy
March 2, 2016 at 3:05 pm
I was working on GPO issues so I was not here when they applied the hotfix. Both Citrix and Microsoft were on the call with the customer so I was not directly involved. I was just asked to write it up.
Webster
March 2, 2016 at 6:03 pm
No problems. I’m just curious. I got hold of the hotfix and tried to run it on a test machine that already had a newer version of ntfs.sys. WUSA said that it wasn’t applicable. Sorry, I’m just intrigued how this hotfix applied.
Cheers,
Jeremy