Citrix XenApp Server Running on Windows Server 2008 R2 Hard Locks Under Heavy Load
On a recent project, the customer experienced an issue where after updating the Citrix Provisioning Services (PVS) image to Internet Explorer (IE) 11 caused a major outage.
What would happen is that is any user or users caused two Microsoft Office applications to run at the same time, the CPU would spike to between 80% to 100% and the server would become unresponsive. The only recourse was to use the hypervisor to either Force Shutdown or Force Reset the server.
The customer’s previous PVS image used the exact same version of Microsoft Office and other core applications. The only thing that changed was an upgrade from IE10 to IE11. After reverting users to other servers for the Office applications, troubleshooting the root cause began.
To make a very long story short, it was NOT:
- An IE11 issue
- A PVS issue
- A XenApp issue
- An Office application issue
- Or a user issue
Turns out there is a known issue on Server 2008 and Server 2008 R2 with the NTFS IO system not being able to handle the stress of large amounts of file IO. Both Citrix and Microsoft were involved in the troubleshooting and reading of dumps. Microsoft sent a hotfix that the support representative said should be installed on every 2008 R2 server whether they experience this issue or not. This has been a known issue since Server 2008.
If this has been a known issue for so long, surely the hotfix is readily available? No it is not. Surely this major bug was fixed in a product update or service pack for Server 2008 or Server 2008 R2? Not it was not. The hotfix is only available by opening a Microsoft support ticket. Good news is Microsoft fixed the bug in Server 2012.
Bottom line was the customer doubled the number of servers being streamed by PVS and upgraded to IE11 and the new users had large mailboxes and extremely large Excel files. The NTFS file system could not keep up with the extra demand and the XenApp 6.5 servers would hard lock when a large Excel file and a large mailbox were opened on the same server.
If you experience this issue, look for A computer that is running Windows Server 2008 R2 SP1 or Windows 7 SP1 freezes under a heavy file I/O load . You know Microsoft, this hotfix is only available from them only if you open a support ticket. I can’t give it to you. 🙂