A few weeks ago, the town I live in had extremely severe thunderstorms roll through. When the tornado sirens went off and all our mobile devices started screaming “Warning: tornado sighted in your immediate area. Take shelter immediately!”, I did not bother doing a proper shutdown of all my lab equipment. The tornado did extensive damage in our town but missed our house by about a mile or two as the crow flies. We were without power most of the night. The power outage lasted longer than the batteries in all my battery backup systems. Needless to say, all my XenServer, vSphere, and vCenter hosts did an ungraceful power off. My two Synology NAS units have USB connections to their UPS and are configured for proper shutdowns in the event of power loss. For the past few weeks, I thought all was well in my lab, but that wasn’t the case I found out.

After the storms passed, I was able to power on all the UPS units and all the XenServer, vSphere and vCenter hosts came to life with no reported errors. It wasn’t until I went to update my XenApp and XenDesktop 7.12 Machine Catalogs that I discovered major issues. None of the catalogs would update. They would go through the process and eventually abort with “no compatible storage sets for secondary disk provisioning” as shown in Figure 1.

I spent quite a bit of time trying to recover from this but was unsuccessful. Since I have just about everything in my lab scripted, I decided to just remove all my “stuff” in Studio and just start over.

#borrowed from Martin Zugec

Write-Host "Removing all applications groups"
Get-BrokerApplicationGroup | Remove-BrokerApplicationGroup

Write-Host "Removing all applications"
Get-BrokerApplication | Remove-BrokerApplication

Write-Host "Removing all Delivery Groups"
Get-BrokerDesktopGroup | Remove-BrokerDesktopGroup -force

Write-Host "Removing all Catalogs"
Get-BrokerCatalog | Remove-BrokerCatalog

Write-Host "Removing all Assignment Policies"
Get-BrokerAssignmentPolicyRule | Remove-BrokerAssignmentPolicyRule

Write-Host "Removing all Access Policies"
Get-BrokerAccessPolicyRule | Remove-BrokerAccessPolicyRule

What I was unable to remove was the Hosting Connection or the Resource Connection. I kept getting an error message saying “There is currently an active background action”. But I had no Applications or Delivery Groups or Machine Catalogs and no Desktops or Machines, how could there be anything considered a background action? After digging around with Get-Command and Get-Help, I eventually (OK it took me two hours) figured out what I needed to do to get rid of the Hosting Connection. I decided to write this article so I could have these steps for reference if I ever run into this fiasco again.

If this were a common occurrence, I would write a script to do all this but this is the first time I have run into this specific issue.

Update 7-Nov-2017: I have written a script to automate this process.

Step 1: Get a list of tasks that are currently running within the MachineCreation Service

PS C:\Users\ctxadmin> Get-ProvTask -active $true
TaskId : a3323a77-25d6-4dbd-9546-4054b202e1c9
Active : True
Host : STUDIO
DateStarted : 12/30/2016 8:27:55 AM
Metadata : {}
CurrentOperation : Running
TaskProgress :
LastUpdateTime : 12/30/2016 9:27:55 AM
ActiveElapsedTime : 3600
DateFinished :
TerminatingError :
Type : DisusedImageCleanUp
Status : Running
ImagesToDelete : {HostingUnitUid '1ed01e13-4694-4f69-ab4f-0692c40a7528', Image 'DiskId
Version=2,LsiLogicScsi,-1,,[VMSSD] ControlDisk-VirtualDiskId-c70c2701-8f21-417c-91d8-b7896831a
f23/ControlDisk-VirtualDiskId-c70c2701-8f21-417c-91d8-b7896831af23.vmdk,resgroup-27,Vmfs :
StorageId datastore-365', Last Processed '12/30/2016 8:11:02 AM', HostingUnitUid
'1ed01e13-4694-4f69-ab4f-0692c40a7528', Image 'DiskId Version=2,LsiLogicScsi,-1,,[VMSSD]
AppDisk-VirtualDiskId-51a5773c-6ee6-4994-bafb-0955f38b7a86/12 Dec 2016
14-16-50.0947Z.vmdk,resgroup-27,Vmfs : StorageId datastore-365', Last Processed '12/30/2016
8:11:22 AM'}
HeartbeatInterval : 01:00:00
RetryInterval : 06:00:00
WorkflowStatus : Running
TaskExpectedCompletion :

Steps 2 and 3: Stop and then Remove the task returned

PS C:\Users\ctxadmin> Stop-ProvTask -TaskId a3323a77-25d6-4dbd-9546-4054b202e1c9
Success

PS C:\Users\ctxadmin> Remove-ProvTask -TaskId a3323a77-25d6-4dbd-9546-4054b202e1c9
Success

Apparently, Get-ProvTask -active $true only returns one task at a time. Repeat Steps 1 through 3 until all tasks are found, stopped, and removed.

PS C:\Users\ctxadmin> Get-ProvTask -active $true
TaskId : d0fbe38b-bd36-4662-b88b-f370bd9b96b0
Active : True
Host : STUDIO
DateStarted : 12/30/2016 10:21:07 AM
Metadata : {}
CurrentOperation : Running
TaskProgress :
LastUpdateTime : 12/30/2016 10:21:07 AM
ActiveElapsedTime : 0
DateFinished :
TerminatingError :
Type : DisusedImageCleanUp
Status : Running
ImagesToDelete : {HostingUnitUid '1ed01e13-4694-4f69-ab4f-0692c40a7528', Image 'DiskId
Version=2,LsiLogicScsi,-1,,[VMSSD] ControlDisk-VirtualDiskId-c70c2701-8f21-417c-91d8-b7896831a
f23/ControlDisk-VirtualDiskId-c70c2701-8f21-417c-91d8-b7896831af23.vmdk,resgroup-27,Vmfs :
StorageId datastore-365', Last Processed '12/30/2016 8:11:02 AM', HostingUnitUid
'1ed01e13-4694-4f69-ab4f-0692c40a7528', Image 'DiskId Version=2,LsiLogicScsi,-1,,[VMSSD]
AppDisk-VirtualDiskId-51a5773c-6ee6-4994-bafb-0955f38b7a86/12 Dec 2016
14-16-50.0947Z.vmdk,resgroup-27,Vmfs : StorageId datastore-365', Last Processed '12/30/2016
8:11:22 AM'}
HeartbeatInterval : 01:00:00
RetryInterval : 06:00:00
WorkflowStatus : Running
TaskExpectedCompletion :

PS C:\Users\ctxadmin> Stop-ProvTask -TaskId d0fbe38b-bd36-4662-b88b-f370bd9b96b0
Success

PS C:\Users\ctxadmin> Remove-ProvTask -TaskId d0fbe38b-bd36-4662-b88b-f370bd9b96b0
Success

PS C:\Users\ctxadmin> Get-ProvTask -active $true
#nothing was return this time

Step 4: Get hosting units.

PS C:\Users\ctxadmin> cd xdhyp:\hostingunits\
PS XDHyp:\hostingunits> dir
PSPath : Citrix.Host.Admin.V2\Citrix.Hypervisor::XDHyp:\hostingunits\VM Network 2
PSParentPath : Citrix.Host.Admin.V2\Citrix.Hypervisor::XDHyp:\hostingunits
PSChildName : VM Network 2
PSDrive : XDHyp
PSProvider : Citrix.Host.Admin.V2\Citrix.Hypervisor
PSIsContainer : True
HostingUnitName : VM Network 2
HostingUnitUid : 1ed01e13-4694-4f69-ab4f-0692c40a7528
HypervisorConnection : vCenter 6 U2
Metadata : {}
MetadataMap : {}
NetworkId : Network:network-368
NetworkPath : XDHyp:\Connections\vCenter 6 U2\WebstersLab.datacenter\XAXDCluster.cluster\VM Network
2.network
PermittedNetworks : {XDHyp:\Connections\vCenter 6 U2\WebstersLab.datacenter\XAXDCluster.cluster\VM Network
2.network}
RootId : domain-c26
RootPath : XDHyp:\Connections\vCenter 6 U2\WebstersLab.datacenter\XAXDCluster.cluster
Storage : {XDHyp:\Connections\vCenter 6 U2\WebstersLab.datacenter\XAXDCluster.cluster\VMSSD.storage}
VMTaggingEnabled : True
UseLocalStorageCaching : False
PersonalvDiskStorage : {XDHyp:\Connections\vCenter 6 U2\WebstersLab.datacenter\XAXDCluster.cluster\PvDSSD.storage}
GpuType : {}
AvailabilityZones : {}
AdditionalStorage : {TemporaryStorage}
CustomProperties :
FullPath :

Steps 5 and 6: Get, Stop and Remove the provisioning tasks associated with the HostingUnitUid.

PS XDHyp:\hostingunits> Get-ProvTask | Where-Object {$_.hostingunit -eq "1ed01e13-4694-4f69-ab4f-0692c40a7528"}|Stop-ProvTask
PS XDHyp:\hostingunits> Get-ProvTask | Where-Object {$_.hostingunit -eq "1ed01e13-4694-4f69-ab4f-0692c40a7528"}|Remove-ProvTask

Steps 7, 8, and 9: Remove the resource connection, the hypervisor connection, and finally the hosting connection.

PS XDHyp:\hostingunits> Remove-Item -path "xdhyp:\HostingUnits\VM Network 2"
PS XDHyp:\hostingunits> Remove-BrokerHypervisorConnection -Name "vCenter 6 U2"
PS XDHyp:\hostingunits> Remove-Item -path "xdhyp:\Connections\vCenter 6 U2"

Now refresh the Hosting node in Studio and the Hosting Connection is gone as shown in Figure 2.

In the south, we don’t mess around when a tornado is spotted very close to where we live. My office and lab are located in the attic of our house. An attic is the LAST place you need to be in a tornado warning. I would rather deal with storage issues than mess around with a tornado. I hope neither even happens again in my lifetime but if either happens, I now know the steps to start over (at least with my lab).

Thanks

Webster