I recently noticed that some of our scheduled automation’s failing with an error message – A general system error occurred: Too many outstanding operations.
The failure happens on some hosts when we use esxcli through PowerShell. It only affects ESXi 8 hosts – both 8.0 and 8U1 hosts. The issue is with an ESXi host because restarting the host resolves the problem for a while.
In vCenter, there is also a log entries in vpxd.log
2023-05-16T07:33:09.491Z error vpxd[04546] [Originator@6876 sub=Default opID=25c7611e] [VpxLRO] -- ERROR lro-33762714 -- 52c4a79f-6dd5-ab87-6ce1-09a069c89e77(52ae3e10-ab1d-348b-e010-c839e5709e02) -- ManagedMethodExecuter-56 -- vmodl.reflect.ManagedMethodExecuter.executeSoap: :vmodl.fault.SystemError
--> Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Too many outstanding operations"
--> msg = "Received SOAP response fault from [<<io_obj p:0x00007f3764b07400, h:140, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-56/vpxa>]: executeSoap
--> Received SOAP response fault from [<<io_obj p:0x000000e087d6bae8, h:26, <TCP '127.0.0.1 : 21548'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: executeSoap
--> A general system error occurred: Too many outstanding operations"
--> }
--> Args:
-->
--> Arg moid:
--> "ha-cli-handler-network-nic"
--> Arg version:
--> "vim25/5.0"
--> Arg method:
--> "vim.EsxCLI.network.nic.get"
--> Arg argument:
--> (vmodl.reflect.ManagedMethodExecuter.SoapArgument) [
--> (vmodl.reflect.ManagedMethodExecuter.SoapArgument) {
--> name = "nicname",
--> val = "<nicname xmlns="urn:vim25">vusb0</nicname>"
--> }
--> ]This error appears in the vpxd.log file several times with different “Arg method” values. From what I can see, all those failing methods are using vim.EsxCLI.
We will continue looking into this issue and open a case with VMware.
