• The Curious Case of the Slow File Transfer

    Let’s get started with a bit of storytelling…

    It all started back in at the end of Q3 2011, I was setting up a new Citrix XenApp 6.5 farm for a migration project at the company I was outsourced at. Back then XenApp 6.5 had been released just a few months earlier.

    Everything was looking fine and the migration was going smoothly. A few weeks after the last user was migrated we began seeing some strange behavior. Most of that was solved by implementing new Citrix Access Gateway (CAG)  VPX’s, but one issue remained: accessing client drive mappings was extremely slow. To make matters worse, the application being used needed to transfer PDF’s (not even large ones, mostly around 200KB) from client drives to the server. Although the copy eventually would always complete, performance was far less than what is considered acceptable. At the beginning of the weirdness, we did see some rare picadm.sys messages in the event log, but they went away later on.

    So after some initial troubleshooting and searching numerous Citrix forums, we came up with the only solution left at that stage: open a support call at Citrix.

    Despite the tests we already performed we were initially met with some skepticism. It took quite some convincing that our problem was very real.

    Enough for storytelling, let’s get to the facts. These are the results of our first round of testing:

    Protocol 10mb c->s 300mb c->s 10mb s->c 300mb s->c
    ICA 1.39 s 45.47 s 2.86 s 92.73 s
    RDP 0.64 s 15.31 s 1.82 s 43.19s

    C->S stands for Client to Server transfer

    S->C stands for Server to Client transfer

    Tests were performed on a LAN-connection, with a firewall between Client and Server.

    Just looking at the numbers, you’ll probably get the level of impact of this bug.

    However, our luck was turning. For the first time, we did have evidence of the problem in our hands.

    After looking at various parameters and tuning, and many months later(!), the problem was identified as being driver-related, and the picadm.sys driver in particular. Yes, that’s right, everything started out with sporadic picadm.sys warnings didn’t it? Citrix Development came up with a fix, actually a double fix: client and server side. They send me private binaries to test with, and the results were amazing.

    First round of tests, done with a 12.1 ICA client:

    Server078 – not patched server

    ——————————————————————————

    1    \\Client\C$\Temp\

    100%        New File             206.6 m        20120501_161341.mp4

    ——————————————————————————

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  206.65 m  206.65 m         0         0         0         0

    Times :   0:04:01   0:04:01                       0:00:00   0:00:00

    Speed :              896474 Bytes/sec.

    Speed :              51.296 MegaBytes/min.

    Server078 – second run (caching?)

    ——————————————————————————

    1    \\Client\C$\Temp\

    100%        New File             206.6 m        20120501_161341.mp4

    ——————————————————————————

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  206.65 m  206.65 m         0         0         0         0

    Times :   0:02:52   0:02:52                       0:00:00   0:00:00

    Speed :             1253267 Bytes/sec.

    Speed :              71.712 MegaBytes/min.

    Server578 – patched server

    ——————————————————————————

    1    \\Client\C$\Temp\

    100%        New File             206.6 m        20120501_161341.mp4

    ——————————————————————————

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  206.65 m  206.65 m         0         0         0         0

    Times :   0:01:25   0:01:25                       0:00:00   0:00:00

    Speed :             2530714 Bytes/sec.

    Speed :             144.808 MegaBytes/min.

    Test copy on this machine using RDP took 22 seconds.

    Second round of tests, with 13.0 ICA client + client patch dll

    Server078 – non patched server

    —————————————————————————–

    1    \\Client\C$\Temp\

    100%        New File             206.6 m        20120501_161341.mp4

    —————————————————————————–

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  206.65 m  206.65 m         0         0         0         0

    Times :   0:00:27   0:00:27                       0:00:00   0:00:00

    Speed :             7938416 Bytes/sec.

    Speed :             454.239 MegaBytes/min.

    Server578 – patched server

    —————————————————————————-

    1    \\Client\C$\Temp\

    100%        New File             206.6 m        20120501_161341.mp4

    —————————————————————————-

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  206.65 m  206.65 m         0         0         0         0

    Times :   0:00:14   0:00:14                       0:00:00   0:00:00

    Speed :            14625740 Bytes/sec.

    Speed :             836.891 MegaBytes/min.

    New run, different file

    Server078 – non patched

    ——————————————————————————

    1    \\Client\C$\Temp\

    100%        New File             197.4 m        App-V-for-RDS-4.6.ISO

    ——————————————————————————

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  197.44 m  197.44 m         0         0         0         0

    Times :   0:00:23   0:00:23                       0:00:00   0:00:00

    Speed :             8757341 Bytes/sec.

    Speed :             501.099 MegaBytes/min.

    Server578 – patched server

    ——————————————————————————

    1    \\Client\C$\Temp\

    100%        New File             197.4 m        App-V-for-RDS-4.6.ISO

    ——————————————————————————

    Total    Copied   Skipped  Mismatch    FAILED    Extras

    Dirs :         1         0         1         0         0         0

    Files :         1         1         0         0         0         0

    Bytes :  197.44 m  197.44 m         0         0         0         0

    Times :   0:00:15   0:00:15                       0:00:00   0:00:00

    Speed :            13271302 Bytes/sec.

    Speed :             759.390 MegaBytes/min.

    Final test round, same set of files used at the start of the case

    300Mb files

    Client->Server                  23s

    Server->Client                  35s

    Conclusion: together, the performance is dramatically improved and up to standards. But even implementing the fix on just one “side”, there’s a benefit.

    The client side fix was scheduled to be part of the Receiver 3.4 release, but at the very last moment didn’t make it. Strangely enough, we did find Receiver 3.4 does improve performance quite considerably.

    The server side fix, which will be part of XA650R01W2K8R2X64055, was originaly scheduled to be released eary January 2013, but it was delayed with another month (who’s counting anyway?)

    BUT… you will not find any mention of this bug/hotfix in the release notes that will be published. Citrix calls this a “transparent” fix because of the client-side component. This actually just means that the server-side fix is included but release notes will not show it until the client-side fix has been released also. The client-side is scheduled for the next Receiver release, early-to-mid march at this time.

    *Update*

    Citrix Support just came back to me… the hotfix is not only “transparent”, but it will be kept private until the client-side fix is released…So I suppose it’s limited and transparent.
    And this brings me to the reason for writing this article. The support case has been ongoing for more than a year, and we’re still not there yet. Everytime around the story changes, so much for “transparent” communication…

    *Update 2*
    Citrix Support confirmed that the client side fix will not be included in the upcoming release of Receiver, due to a lot of regressionn issues. The fix is now scheduled to be included in the release after that. But… Receiver 3.5 should have been among us already, so who knows what will be included in the next (Excalibur release?) Receiver?

    Another surprise… on april 30th, Citrix released XA650R01W2K8R2X64075, fixing picadm.sys stuff and superseeding XA650R01W2K8R2X64028. Wait a minute… XA650R01W2K8R2X64055 also superseeded XA650R01W2K8R2X64028. So after checking Citrix Support confirmed that XA650R01W2K8R2X64075 does indeed superseed XA650R01W2K8R2X64055. So the fix Citrix is keeping private is already replaced by a public fix… I can’t help to wonder if  XA650R01W2K8R2X64055 will ever see light of day. Or even if we will ever see this bug appearing in release notes of any hotfix for that matter…

    Citrix Support also confirmed that all fixes will be included in the HRP2, out soon…

    ,

    About Bart Jacobs

    Bart Jacobs is a Senior System Engineer/Consultant based in Belgium. He started his career back in 1998. One of the first projects he worked on in those days was Citrix Metaframe 1.8 on Microsoft Windows NT 4 Terminal Server codename "Hydra". Over the years, Citrix technology has always been a major theme in his professional career, resulting in becoming a true technical expert in the matter. In the last few years, he has also become an expert in virtualization technology, with a special interest in a real challenger in this business: Citrix XenServer. Bart has founded his own company BJ IT back in 2007 and is mainly working as a (Citrix) consultant now. In 2019, Bart received his Citrix CTA award.

    View all posts by Bart Jacobs

    12 Responses to “The Curious Case of the Slow File Transfer”

    1. Enterprise IT Says:

      I was curious if you have had this issue with XenDesktop 7.1? I’m getting about 1MB/s file transfer through the Citrix Receiver from a locally mapped drive. Using Receiver 4.1. Being a newer product, I had hoped the fixes described here would have been included in XenDesktop 7.1 however perhaps not.

      Thanks,

      Chris

      Reply

    2. Steve Says:

      Bart, Any update to your Jan 27th. post related to RO3 and picadm.sys? We’ve been struggling with this same issue for over a year now with no fix. Have Citrix engaged however not received any private fixes. We are at RO3 and has not resolved itself.
      File version – 6.2.7500.32 Date – 11/18/13
      Thanks in advance!

      Reply

    3. jason Says:

      Also, what was the event id for the picadm.sys error that you saw?

      thanks again

      Reply

    4. jason Says:

      Any updates? we are HRP3 and have tried 3.4 recover (unreleased update 3) with the same issues. Most of our issues are with CDrom transfers. Disk/Usb transfers do not seem to suffer so much.

      Would you mind sharing your case # so I can inform the person I am working with that someone else has the same issue.

      thanks

      Reply

    5. chris Says:

      Hi,
      Also searching for an confirmed solution, found this but don’t know if it is the solution for this;

      http://social.technet.microsoft.com/Forums/en/w7itpronetworking/thread/f79b0ff2-9f1b-464c-9664-5595fc3556d2

      Reply

    6. Ryan Gallier Says:

      Receiver 4.0 just came out. I see significant improvement on my XA6.5 R02 systems. Whats the deal with those of us who still have 6.0 farms where just “upgrading” isn’t really an option?

      Reply

    7. Andrew Parry Says:

      Carl,

      any news on whether a version of Receiver has been released that resolves this? And also I see HFRP02 has been released as a beta. Any ideas when the public full release will be..they say Q2 2013 but when will it really be? and are you confident it WILL include the fixes from XA650R01W2K8R2X64055, to resolve picadm.sys issues..

      Cheers,

      Andy

      Reply

      • Bart Jacobs Says:

        Andy,

        no receiver was released until now that resolves this issue. HRP2/FP2 for XenApp 6.5 will be lauched in June 2013, together with XenDesktop 7. XA650R01W2K8R2X64055 is already superseeded by XA650R01W2K8R2X64075.
        HRP2/FP2 should include this fix, and I’m actually working on a little follow up article in regards to developments in FP2.

        Reply

        • Julien Sybille Says:

          Hello any update about this curious case ?

          I’m actually making test with XenApp6.5 RP3 & Receiver 4.1 and It’s look like I steel have the problem or may I miss something ?

          Reply

          • Bart Jacobs Says:

            Well… HRP3 contains a version of picadm.sys that causes some systems to BDOD on reboot. Apparently, Citrix has a new version in the works again… so, stay tuned!

            Reply

    Leave a Reply