VCF Upgrade Issue

I was performing the VCF upgrade from 4.5 to 4.5.1, I faced an issue, where the host evacuation during MaintenanceMode was stuck at 17%.

As per the VCF upgrade procedure, SDDC and vRSLCM upgrades were completed successfully and the next upgrade to be performed was NSX-T, during the NSX-T upgrade, there is a task where the NSX-T vibs will be upgraded on all the ESXi hosts in the management cluster.

The NSX vib upgrade started on ESXi01 and the host started entering MaintenanceMode and was stuck at 17% the task on SDDC UI was showing as running after like few hours the upgrade failed with the below error:

Checked the vpxd logs in vCenter and could see the below error pointing the issue toward the vds portgroup issue:

vpxd.log

2023-07-26T09:15:15.437+01:00 info vpxd[20808] [Originator@6876 sub=vpxLro opID=lro-16305299-6cc076d8-01-01] [VpxLRO] -- FINISH lro-16305301
2023-07-26T09:15:15.437+01:00 info vpxd[20808] [Originator@6876 sub=Default opID=lro-16305299-6cc076d8-01-01] [VpxLRO] -- ERROR lro-16305301 --  -- VmprovWorkflow: vim.fault.NotFound:
--> Result:
--> (vim.fault.NotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vpxd.dvs.portNotFound.label",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "port",
-->                value = "149"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ]
-->    msg = ""
--> }
--> Args:
-->
2023-07-26T09:15:15.438+01:00 error vpxd[20808] [Originator@6876 sub=drsExec opID=lro-16305299-6cc076d8-01] Failed migrating VM [vim.VirtualMachine:vm-49,vcf] to host vim.HostSystem:host-54
2023-07-26T09:15:15.438+01:00 info vpxd[20808] [Originator@6876 sub=vpxLro opID=lro-16305299-6cc076d8-01] [VpxLRO] -- FINISH task-203368

Even the ESXi01 hostd and vpxa logs point to the same issue.

vpxa.log

2023-07-26T06:47:23.647Z info vpxa[2103319] [Originator@6876 sub=Default opID=adb36f2] [VpxLRO] -- ERROR lro-555363 -- hostDvsManager -- vim.dvs.HostDistributedVirtualSwitchManager.deletePorts: vim.fault.PlatformConfigFault:
--> Result:
--> (vim.fault.PlatformConfigFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.esx.hostctl.default",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "reason",
-->                value = "Unable to delete DVPort "" that is in use, use list: vcf.eth0"
-->             }
-->          ],
-->          message = "Operation failed, diagnostics report: Unable to delete DVPort "" that is in use, use list: vcf.eth0"
-->       }
-->    ],
-->    text = ""
-->    msg = "Received SOAP response fault from [<<io_obj p:0x0000005ea83ae7c8, h:18, <TCP '127.0.0.1 : 47025'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: deletePorts
--> An error occurred during host configuration."
--> }
--> Args:
-->
--> Arg switchUuid:
--> "50 3a 0a d1 b5 40 a8 83-6b 7d 54 fc c8 a0 44 07"
--> Arg portSpec:
--> (vim.dvs.HostDistributedVirtualSwitchManager.PortDeleteSpec) [
-->   (vim.dvs.HostDistributedVirtualSwitchManager.PortDeleteSpec) {
-->       portKey = "149",
-->       deletePortFile = true,
-->       systemCleanup = true
-->    }
--> ]

hostd.log

2023-07-26T08:22:24.746Z info hostd[2103573] [Originator@6876 sub=Hostsvc.NetworkProvider opID=44762013-2613 user=vpxuser] deleted port file /vmfs/volumes/vsan:52142dadc1c0873a-666d2b1e213b3b5/db165264-f04a-1276-4fd7-e8ebd3709f6e/.dvsData/50 3a 0a d1 b5 40 a8 83-6b 7d 54 fc c8 a0 44 07/149 : 149
2023-07-26T08:22:24.746Z warning hostd[2103573] [Originator@6876 sub=Hostsvc.NetworkProvider opID=44762013-2613 user=vpxuser] Error deleting dvport 50 3a 0a d1 b5 40 a8 83-6b 7d 54 fc c8 a0 44 07-149 : Unable to delete DVPort "" that is in use, use list: vcf.eth0
2023-07-26T08:22:24.747Z info hostd[2103573] [Originator@6876 sub=AdapterServer opID=44762013-2613 user=vpxuser] AdapterServer caught exception; <<52d5b718-3292-79db-4f66-9423b22328be, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 24250'>>, ha-hostdvsmanager, vim.dvs.HostDistributedVirtualSwitchManager.deletePorts>, N7Hostsvc21HaPlatformConfigFault9ExceptionE(Fault cause: vim.fault.PlatformConfigFault
--> )
--> [context]zKq7AVICAgAAANQIPgENaG9zdGQAANJCF2xpYnZtYWNvcmUuc28AAYhiT2hvc3RkAAEQiKcBs/inAcjqnAGFtZSCAWoyAWxpYnZpbS10eXBlcy5zbwAB2pZSAMzHLQBUAy4AAhE/Azt9AGxpYnB0aHJlYWQuc28uMAAEbdEObGliYy5zby42AA==[/context]
2023-07-26T08:22:24.751Z info hostd[2103573] [Originator@6876 sub=Solo.Vmomi opID=44762013-2613 user=vpxuser] Activation finished; <<52d5b718-3292-79db-4f66-9423b22328be, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 24250'>>, ha-hostdvsmanager, vim.dvs.HostDistributedVirtualSwitchManager.deletePorts>
2023-07-26T08:22:24.751Z verbose hostd[2103573] [Originator@6876 sub=Solo.Vmomi opID=44762013-2613 user=vpxuser] Arg switchUuid:
--> "50 3a 0a d1 b5 40 a8 83-6b 7d 54 fc c8 a0 44 07"
2023-07-26T08:22:24.751Z verbose hostd[2103573] [Originator@6876 sub=Solo.Vmomi opID=44762013-2613 user=vpxuser] Arg portSpec:
--> (vim.dvs.HostDistributedVirtualSwitchManager.PortDeleteSpec) [
-->   (vim.dvs.HostDistributedVirtualSwitchManager.PortDeleteSpec) {
-->       portKey = "149",
-->       deletePortFile = true,
-->       systemCleanup = true
-->    }
--> ]
2023-07-26T08:22:24.752Z info hostd[2103573] [Originator@6876 sub=Solo.Vmomi opID=44762013-2613 user=vpxuser] Throw vim.fault.PlatformConfigFault

This issue happens when the ESXi host is added into vDS and perform a VM migration, and the VM on that particular Host does not register on dvport information correctly and there is a mismatch in the vDS db,

I found the below two workarounds to continue with the upgrade activity:

First:

• On your vCenter UI navigate to Networking --> Select the dVSwitch --> Create a new dummy portgroup.

• Then assign the newly created portgroup to the problematic VM (which is failing to migrate) from the Edit settings of the VM.

• Once the save is completed successfully then revert back the VM to its original portgroup.

Second:

• Edit the VM setting from the vCenter and change the VM portgroup to any other available portgroup and save the settings.

• Once the save is completed successfully then revert back the VM to its original portgroup.

Once any of the above workarounds are applied I restarted the upgrade task from SDDC and this time host entered the MaintenanceMode successfully without any further issues.