It all started back in at the end of Q3 2011, I was setting up a new Citrix XenApp 6.5 farm for a migration project at the company I was outsourced at. Back then XenApp 6.5 had been released just a few months earlier.

Everything was looking fine and the migration was going smoothly. A few weeks after the last user was migrated we began seeing some strange behavior. Most of that was solved by implementing new Citrix Access Gateway (CAG)  VPX’s, but one issue remained: accessing client drive mappings was extremely slow. To make matters worse, the application being used needed to transfer PDF’s (not even large ones, mostly around 200KB) from client drives to the server. Although the copy eventually would always complete, performance was far less than what is considered acceptable. At the beginning of the weirdness, we did see some rare picadm.sys messages in the event log, but they went away later on.

So after some initial troubleshooting and searching numerous Citrix forums, we came up with the only solution left at that stage: open a support call at Citrix.

Despite the tests we already performed we were initially met with some skepticism. It took quite some convincing that our problem was very real.

Enough for storytelling, let’s get to the facts. These are the results of our first round of testing:

Protocol	10mb c->s	300mb c->s	10mb s->c	300mb s->c
ICA	1.39 s	45.47 s	2.86 s	92.73 s
RDP	0.64 s	15.31 s	1.82 s	43.19s

C->S stands for Client to Server transfer

S->C stands for Server to Client transfer

Tests were performed on a LAN-connection, with a firewall between Client and Server.

Just looking at the numbers, you’ll probably get the level of impact of this bug.

However, our luck was turning. For the first time, we did have evidence of the problem in our hands.

After looking at various parameters and tuning, and many months later(!), the problem was identified as being driver-related, and the picadm.sys driver in particular. Yes, that’s right, everything started out with sporadic picadm.sys warnings didn’t it? Citrix Development came up with a fix, actually a double fix: client and server side. They send me private binaries to test with, and the results were amazing.

First round of tests, done with a 12.1 ICA client:

Server078 – not patched server

——————————————————————————

1    \\Client\C$\Temp\

100%        New File             206.6 m        20120501_161341.mp4

——————————————————————————

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  206.65 m  206.65 m         0         0         0         0

Times :   0:04:01   0:04:01                       0:00:00   0:00:00

Speed :              896474 Bytes/sec.

Speed :              51.296 MegaBytes/min.

Server078 – second run (caching?)

——————————————————————————

1    \\Client\C$\Temp\

100%        New File             206.6 m        20120501_161341.mp4

——————————————————————————

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  206.65 m  206.65 m         0         0         0         0

Times :   0:02:52   0:02:52                       0:00:00   0:00:00

Speed :             1253267 Bytes/sec.

Speed :              71.712 MegaBytes/min.

Server578 – patched server

——————————————————————————

1    \\Client\C$\Temp\

100%        New File             206.6 m        20120501_161341.mp4

——————————————————————————

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  206.65 m  206.65 m         0         0         0         0

Times :   0:01:25   0:01:25                       0:00:00   0:00:00

Speed :             2530714 Bytes/sec.

Speed :             144.808 MegaBytes/min.

Test copy on this machine using RDP took 22 seconds.

Second round of tests, with 13.0 ICA client + client patch dll

Server078 – non patched server

—————————————————————————–

1    \\Client\C$\Temp\

100%        New File             206.6 m        20120501_161341.mp4

—————————————————————————–

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  206.65 m  206.65 m         0         0         0         0

Times :   0:00:27   0:00:27                       0:00:00   0:00:00

Speed :             7938416 Bytes/sec.

Speed :             454.239 MegaBytes/min.

Server578 – patched server

—————————————————————————-

1    \\Client\C$\Temp\

100%        New File             206.6 m        20120501_161341.mp4

—————————————————————————-

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  206.65 m  206.65 m         0         0         0         0

Times :   0:00:14   0:00:14                       0:00:00   0:00:00

Speed :            14625740 Bytes/sec.

Speed :             836.891 MegaBytes/min.

New run, different file

Server078 – non patched

——————————————————————————

1    \\Client\C$\Temp\

100%        New File             197.4 m        App-V-for-RDS-4.6.ISO

——————————————————————————

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  197.44 m  197.44 m         0         0         0         0

Times :   0:00:23   0:00:23                       0:00:00   0:00:00

Speed :             8757341 Bytes/sec.

Speed :             501.099 MegaBytes/min.

Server578 – patched server

——————————————————————————

1    \\Client\C$\Temp\

100%        New File             197.4 m        App-V-for-RDS-4.6.ISO

——————————————————————————

Total    Copied   Skipped  Mismatch    FAILED    Extras

Dirs :         1         0         1         0         0         0

Files :         1         1         0         0         0         0

Bytes :  197.44 m  197.44 m         0         0         0         0

Times :   0:00:15   0:00:15                       0:00:00   0:00:00

Speed :            13271302 Bytes/sec.

Speed :             759.390 MegaBytes/min.

Final test round, same set of files used at the start of the case

300Mb files

Client->Server                  23s

Server->Client                  35s

Conclusion: together, the performance is dramatically improved and up to standards. But even implementing the fix on just one “side”, there’s a benefit.

The client side fix was scheduled to be part of the Receiver 3.4 release, but at the very last moment didn’t make it. Strangely enough, we did find Receiver 3.4 does improve performance quite considerably.

The server side fix, which will be part of XA650R01W2K8R2X64055, was originaly scheduled to be released eary January 2013, but it was delayed with another month (who’s counting anyway?)

BUT… you will not find any mention of this bug/hotfix in the release notes that will be published. Citrix calls this a “transparent” fix because of the client-side component. This actually just means that the server-side fix is included but release notes will not show it until the client-side fix has been released also. The client-side is scheduled for the next Receiver release, early-to-mid march at this time.

*Update*

Citrix Support just came back to me… the hotfix is not only “transparent”, but it will be kept private until the client-side fix is released…So I suppose it’s limited and transparent.
And this brings me to the reason for writing this article. The support case has been ongoing for more than a year, and we’re still not there yet. Everytime around the story changes, so much for “transparent” communication…

*Update 2*
Citrix Support confirmed that the client side fix will not be included in the upcoming release of Receiver, due to a lot of regressionn issues. The fix is now scheduled to be included in the release after that. But… Receiver 3.5 should have been among us already, so who knows what will be included in the next (Excalibur release?) Receiver?

Another surprise… on april 30th, Citrix released XA650R01W2K8R2X64075, fixing picadm.sys stuff and superseeding XA650R01W2K8R2X64028. Wait a minute… XA650R01W2K8R2X64055 also superseeded XA650R01W2K8R2X64028. So after checking Citrix Support confirmed that XA650R01W2K8R2X64075 does indeed superseed XA650R01W2K8R2X64055. So the fix Citrix is keeping private is already replaced by a public fix… I can’t help to wonder if  XA650R01W2K8R2X64055 will ever see light of day. Or even if we will ever see this bug appearing in release notes of any hotfix for that matter…

Citrix Support also confirmed that all fixes will be included in the HRP2, out soon…