Home > Blogs > Support Insider > Author Archives: VMware

Author Archives: VMware

A look at All Paths Down in vSphere

Karthick SivaramakrishnanToday we have a guest post from Karthick Sivaramakrishnan, who is a 3 year veteran at VMware. His primary field of expertise is vSphere Storage and Site Recovery Manager.

This blog post is centered around how ESXi handles unscheduled storage disconnects on vSphere 5.x and 6.x. An unscheduled storage disconnect means some issue in the vSphere environment has led to All-Paths-Down (APD) for a datastore.  An APD situation will be seen when ESXi host does not have any path to communicate with a lun on the storage array.

ESXi host can encounter an APD under several conditions. As a result, we may end up having VMs running on a given datastore go down, the host could get disconnected from vCenter, and in worst cases ESXi could become unresponsive.

From vSphere version 5.x and onwards, we are able to discern whether a disconnect is permanent or transient. Ideally a transient disconnect leads to All Paths Down state and ESXi expects the device to have a temporary disconnect. When we see permanent device loss or PDL the device is expected to have a non-recoverable issue like a hardware error or the lun is unmapped.

In the below example we see all iSCSI datastores are in inactive state.

Datastores

To determine what caused this issue we see ESXi logs, particularly vmkernel and vobd. This issue will be evident in the vmkernel logs.

vmkernel log

2017-01-10T13:04:26.803Z cpu1:32896)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6000eb31dffdc33a0000000000000028] has entered the All Paths Down state.

2017-01-10T13:04:26.818Z cpu0:32896)StorageApdHandlerEv: 110: Device or filesystem with identifier [naa.6000eb31dffdc33a000000000000002a] has entered the All Paths Down state.

vobd log

2017-01-10T13:04:26.905Z: [scsiCorrelator] 475204262us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.6000eb31dffdc33a0000000000000028. Path vmhba33:C0:T1:L0 is down. Affected datastores: “Green”.

2017-01-10T13:04:26.905Z: [scsiCorrelator] 475204695us: [esx.problem.storage.connectivity.lost] Lost connectivity to storage device naa.6000eb31dffdc33a000000000000002a. Path vmhba33:C0:T0:L0 is down. Affected datastores: “Grey”.

From these logs we understand that ESXi host has lost connectivity to the datastore. Any virtual machines using the affected datastore may become unresponsive. In this example while the datastores was mounted on ESXi, we lost the network uplink on the nic that was used for iSCSI connection. This was a transient issue and the datastore came up once the network uplink was restored.

In the below example we see Datastore Black is in inactive state.

Datastore view missing

If we look into the logs to determine whats going on we see these events.

Vmkernel.log

2017-01-09T12:42:09.365Z cpu0:32888)ScsiDevice: 6878: Device naa.6000eb31dffdc33a0000000000000063 APD Notify PERM LOSS; token num:1

2017-01-09T12:42:09.366Z cpu1:32916)StorageApdHandler: 1066: Freeing APD handle 0x430180b88880 [naa.6000eb31dffdc33a0000000000000063]

2017-01-09T12:49:01.260Z cpu1:32786)WARNING: NMP: nmp_PathDetermineFailure:2973: Cmd (0xc1) PDL error (0x5/0x25/0x0) – path vmhba33:C0:T3:L0 device naa.6000eb31dffdc33a0000000000000063 – triggering path evaluation

2017-01-09T12:49:01.260Z cpu1:32786)ScsiDeviceIO: 2651: Cmd(0x439d802ec580) 0xfe, CmdSN 0x4b7 from world 32776 to dev “naa.6000eb31dffdc33a0000000000000063” failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.

2017-01-09T12:49:01.300Z cpu0:40210)WARNING: NMP: vmk_NmpSatpIssueTUR:1043: Device naa.6000eb31dffdc33a0000000000000063 path vmhba33:C0:T3:L0 has been unmapped from the array

After some time passes you will see this message:

2017-01-09T13:13:11.942Z cpu0:32872)ScsiDevice: 1718: Permanently inaccessible device :naa.6000eb31dffdc33a0000000000000063 has no more open connections. It is now safe to unmount datastores (if any) and delete the device.

In this case the lun was unmapped from the array for this host and that is not a transient issue. Sens data 0x5 0x25 0x0 corresponds to “LOGICAL UNIT NOT SUPPORTED” which indicates the device is in Permanent Device Loss (PDL) state. Once ESXi knows the device is in PDL state it does not wait for the device to return back.

ESXi only checks ASC/ASCQ and if it happens to be 0x25/0x0 or  0x68/0x0, it marks device as PDL.

VMware KB 2004684 has in-depth information around APD and PDL situations. It also talks about planned and unplanned PDL. You can read it here: Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and 6.x (2004684)

Further on in the hostd logs you will see some additional events that will correlate to storage connection.  Look for the below event id’s.

Event ID : esx.problem.storage.connectivity.lost

datestores3

“esx.problem.storage.connectivity.lost” event indicates a loss in connectivity to the specified storage device.  Any virtual machines using the affected datastore may become unresponsive.

Event ID : esx.problem.scsi.device.state.permanentloss

datastores4

“esx.problem.scsi.device.state.permanentloss” event indicates a permanent device loss.

Inconsistencies between the IDP hostname and the vRA hostname producing behavior and authentication issues in vRealize Automation 7.x

Questions:

  • Have you been getting certificate trust issues when logging into vRealize Automation 7.x only to find that the certificate is trusted once you are logged in?
  • Have you ever noticed that your login page for vRA 7.x uses a different host name in the URL than the application?
  • Have you been having difficulty logging into the embedded vRealize Orchestrator instance?
  • Have you been trying to use vRealize CodeStream only to find unusual authentication issues?

It could be that you have a simple misconfiguration in your deployment that can be easily corrected. That is the subject of this blog post.

For many deployments, a fully qualified domain name (FQDN) will be selected to access your vRealize Automation (vRA) 7.x application that is different than the actual host name of the box. This alias could be created for a simple installation to make it easier for users to recall the URL or it could be used to access the VIP provided by a load balancer in a more distributed installation.

Depending upon the steps taken during the deployment (e.g., changing the vRA FQDN after the installation), the hostname in the IDP can be different than hostname configured for use with vRA. For example, the following image illustrates the hostname configured for accessing the vRA application in a hands-on lab:

vRealize Hostname

However, after appropriate tinkering, when users attempt to log in, they see the following:

Connection is Untrusted

In order to correct this, take appropriate backups then log in as a user with permission to administer the directory configuration (i.e., a tenant administrator). Select the Administration tab and navigate to Directories Management > Identity Providers. Select an IDP and you will be presented with an interface similar to the following:

ldP Hostname

Take note of the hostname and adjust this to be consistent with the vRA application FQDN found in the administration interface (VAMI) above. It is particularly important to rectify this if you happen to have the short name in the IDP, which can cause a number of problems. The changes take place immediately without restarting any services, though there may be additional steps needed to correct a particular issue. For example, I have had to re-register the vRO authentication after correcting this due to the state it was in after troubleshooting.

For additional reading, please review the following KB articleVMware KB Logging in to embedded vRealize Orchestrator fails (2146063)

Path failover may not be successful when using Cisco MDS Switches on NX-OS 7.3 and FCoE based HBAs

So I wanted to get this blog post out sooner rather than later as it might effect a significant number of customers. In a nutshell, if you perform array maintenance that requires you to reboot a storage controller, the probability of successful path failover is low. This is effectively due to stale entries in the fiber channel name server on Cisco MDS switches running NX-OS 7.3, which is a rather new code release. As the title suggests, this only affects FCoE HBAs, specifically ones that rely on our libfc/libfcoe stack for FCoE connectivity. Such HBAs would be Cisco fnic HBAs as well as a handful of Emulex FCoE HBAs and a couple others.

Here is an example of a successful path failover after receiving an RSCN (Register State Change Notification) from the array controller after performing a reboot:

2016-07-07T17:36:34.230Z cpu17:33461)<6>host4: disc: Received an RSCN event
 2016-07-07T17:36:34.230Z cpu17:33461)<6>host4: disc: Port address format for port (e50800)
 2016-07-07T17:36:34.230Z cpu17:33461)<6>host4: disc: RSCN received: not rediscovering. redisc 0 state 9 in_prog 0
 2016-07-07T17:36:34.231Z cpu14:33474)<6>host4: disc: GPN_ID rejected reason 9 exp 1
 2016-07-07T17:36:34.231Z cpu14:33474)<6>host4: rport e50800: Remove port
 2016-07-07T17:36:34.231Z cpu14:33474)<6>host4: rport e50800: Port entered LOGO state from Ready state
 2016-07-07T17:36:34.231Z cpu14:33474)<6>host4: rport e50800: Delete port
 2016-07-07T17:36:34.231Z cpu54:33448)<6>host4: rport e50800: work event 3
 2016-07-07T17:36:34.231Z cpu54:33448)<7>fnic : 4 :: fnic_rport_exch_reset called portid 0xe50800
 2016-07-07T17:36:34.231Z cpu54:33448)<7>fnic : 4 :: fnic_rport_reset_exch: Issuing abts
 2016-07-07T17:36:34.231Z cpu54:33448)<6>host4: rport e50800: Received a LOGO response closed
 2016-07-07T17:36:34.231Z cpu54:33448)<6>host4: rport e50800: Received a LOGO response, but in state Delete
 2016-07-07T17:36:34.231Z cpu54:33448)<6>host4: rport e50800: work delete

Here is a breakdown of what you just read:

  1. RSCN is received from the array controller
  2. Operation is now is state = 9
  3. GPN_ID (Get Port Name ID) is issued to the switches but is rejected because the state is 9 (See http://lists.open-fcoe.org/pipermail/fcoe-devel/2009-June/002828.html)
  4. LibFC begins to remove the port information on the host
  5. Port enters LOGO (Logout) state from previous state, which was Ready
  6. LibFC Deletes the port information

After this the ESX host will failover to other available ports, which would be on the peer SP:

2016-07-07T17:36:44.233Z cpu33:33459)<3> rport-4:0-1: blocked FC remote port time out: saving binding
 2016-07-07T17:36:44.233Z cpu55:33473)<7>fnic : 4 :: fnic_terminate_rport_io called wwpn 0x524a937aeb740513, wwnn0xffffffffffffffff, rport 0x0x4309b72f3c50, portid 0xffffffff
 2016-07-07T17:36:44.257Z cpu52:33320)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x2a (0x43a659d15bc0, 36277) to dev "naa.624a93704d1296f5972642ea0001101c" on path "vmhba3:C0:T0:L1" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:FAILOVER

A Host status of H:0x1 means NO_CONNECT, hence the failover.

Now here is an example of the same operation on a Cisco MDS switch running NX-OS 7.3 when a storage controller on the array is rebooted:

2016-07-14T19:02:03.551Z cpu47:33448)<6>host2: disc: Received an RSCN event
 2016-07-14T19:02:03.551Z cpu47:33448)<6>host2: disc: Port address format for port (e50900)
 2016-07-14T19:02:03.551Z cpu47:33448)<6>host2: disc: RSCN received: not rediscovering. redisc 0 state 9 in_prog 0
 2016-07-14T19:02:03.557Z cpu47:33444)<6>host2: rport e50900: ADISC port
 2016-07-14T19:02:03.557Z cpu47:33444)<6>host2: rport e50900: sending ADISC from Ready state
 2016-07-14T19:02:23.558Z cpu47:33448)<6>host2: rport e50900: Received a ADISC response
 2016-07-14T19:02:23.558Z cpu47:33448)<6>host2: rport e50900: Error 1 in state ADISC, retries 0
 2016-07-14T19:02:23.558Z cpu47:33448)<6>host2: rport e50900: Port entered LOGO state from ADISC state
 2016-07-14T19:02:43.560Z cpu2:33442)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:02:43.560Z cpu2:33442)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:02:43.560Z cpu58:33446)<6>host2: rport e50900: Port entered LOGO state from LOGO state
 2016-07-14T19:03:03.563Z cpu54:33449)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:03:03.563Z cpu54:33449)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:03:03.563Z cpu2:33442)<6>host2: rport e50900: Port entered LOGO state from LOGO state
 2016-07-14T19:03:23.565Z cpu32:33447)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:03:23.565Z cpu32:33447)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:03:23.565Z cpu54:33449)<6>host2: rport e50900: Port entered LOGO state from LOGO state
 2016-07-14T19:03:43.567Z cpu50:33445)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:03:43.567Z cpu50:33445)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:03:43.567Z cpu32:33447)<6>host2: rport e50900: Port entered LOGO state from LOGO state
 2016-07-14T19:04:03.568Z cpu54:33443)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:04:03.568Z cpu54:33443)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:04:03.569Z cpu32:33472)<6>host2: rport e50900: Port entered LOGO state from LOGO state
 2016-07-14T19:04:43.573Z cpu20:33473)<6>host2: rport e50900: Received a LOGO response timeout
 2016-07-14T19:04:43.573Z cpu20:33473)<6>host2: rport e50900: Error -1 in state LOGO, retrying
 2016-07-14T19:04:43.573Z cpu54:33443)<6>host2: rport e50900: Port entered LOGO state from LOGO state

Notice the difference? Here is a breakdown of what happened this time:

  1. RSCN is received from the array controller
  2. Operation is now is state = 9
  3. GPN_ID (Get Port Name ID) is issued to the switches but is NOT rejected
  4. Since GPN_ID is valid, LibFC issues an Address Discovery (ADISC)
  5. 20 seconds later the ADISC sent times out and this continues to occur every 20 seconds

The problem is that the ADISC will continue this behavior until the array controller completes the reboot and is back online:

2016-07-14T19:04:47.276Z cpu56:33451)<6>host2: disc: Received an RSCN event
 2016-07-14T19:04:47.276Z cpu56:33451)<6>host2: disc: Port address format for port (e50900)
 2016-07-14T19:04:47.276Z cpu56:33451)<6>host2: disc: RSCN received: not rediscovering. redisc 0 state 9 in_prog 0
 2016-07-14T19:04:47.277Z cpu20:33454)<6>host2: rport e50900: Login to port
 2016-07-14T19:04:47.277Z cpu20:33454)<6>host2: rport e50900: Port entered PLOGI state from LOGO state
 2016-07-14T19:04:47.278Z cpu57:33456)<6>host2: rport e50900: Received a PLOGI accept
 2016-07-14T19:04:47.278Z cpu57:33456)<6>host2: rport e50900: Port entered PRLI state from PLOGI state
 2016-07-14T19:04:47.278Z cpu52:33458)<6>host2: rport e50900: Received a PRLI accept
 2016-07-14T19:04:47.278Z cpu52:33458)<6>host2: rport e50900: PRLI spp_flags = 0x21
 2016-07-14T19:04:47.278Z cpu52:33458)<6>host2: rport e50900: Port entered RTV state from PRLI state
 2016-07-14T19:04:47.278Z cpu57:33452)<6>host2: rport e50900: Received a RTV reject
 2016-07-14T19:04:47.278Z cpu57:33452)<6>host2: rport e50900: Port is Ready

What is actually happening here is that the Cisco MDS switches are quick to receive the RSCN from the array controller and pass it along to the host HBAs however due to a timing issue the entries for that array controller in the FCNS (Fiber Channel Name Server) database are still present when the host HBAs issue the GPN_ID so the switches respond to that request instead of rejecting it. If you review the entry in http://lists.open-fcoe.org/pipermail/fcoe-devel/2009-June/002828.html you see that code was added to validate that the target is actually off the fabric instead of assuming it would be by the RSCN alone. There are various reasons to do this but suffice it to say that it is better to be safe than sorry in this instance.

Unfortunately there is no fix for this at this time, which is why this is potentially so impactful to our customers since it means they effectively are unable to perform array maintenance without the risk of VMs crashing or even corruption. Cisco is fixing this in 7.3(1), which due out in a few weeks.

Here are a couple of references regarding this issue:

 

Cheers,
Nathan Small
Technical Director
Global Support Services
VMware

NSXv License Versions

In NSX 6.2.2 VMware introduced 3 different license editions; Standard, Advanced, and Enterprise. These license editions allow you align NSX with your company’s use case.

Standard Edition: Automates IT workflows, bringing agility to the data center network and reducing network operating costs and complexity.

Advanced Edition: Standard Edition plus a fundamentally more secure data center with micro-segmentation. Helps secure the data center to the highest levels, while automating IT provisioning of security.

Enterprise Edition: Advanced Edition plus networking and security across multiple domains. Enables the data center network to extend across multiple sites and connect to high-throughput physical workloads.

new nsx offerings

If you are an existing NSX customer the upgrade to 6.2.4 will not impact your license type and components you are entitled to.

For vCNS customers you can upgrade to NSX, however when doing so you will get an Endpoint for NSX only license. You can enable the additional features by purchasing the NSX license that fits your business requirements.

installation and upgrade

License Capacity Usage

NSX capacity usage calculation method only reports for clusters prepared and enabled with DFW and VXLAN. CPU count is number of CPUs (sockets) of all prepared hosts. VM count and Concurrent Users is the count of all powered on VMs in the cluster. This VM count does not include system VMs (service VMs, partner VMs, edge appliances etc).

NSX usage is reported correctly under the NSX Manager in NSX vSphere Webclient Plugin. Please note under license management in VC the NSX license will report Usage as ZERO.

nsx usage

vShield Endpoint License in NSX 6.2.4

vShield Endpoint is a component of vCloud Network and Security (vCNS). This component allows you to offload antivirus and anti-malware agent processing to a dedicated secure virtual appliance. With the release of NSX 6.2.4 the default license is NSX for vShield Endpoint allowing you to manage you vShield Endpoint environment with NSX. Customers who purchased vSphere with vShield Endpoint (Essential Plus and above) will be able to download NSX. This means that NSX will appear on the vSphere download site, just like vCNS does today. To ensure customers do not use any other unlicensed NSX features (eg. VXLAN, DFW, Edge services), the license key will have hard enforcement to prevent NSX host preparation and block Edge creation. If you require an evaluation license key, please request this through VMware sales.

 

Log Insight License in NSX 6.2.2 and above

VMware vRealize Log Insight 3.3.2 for NSX provides intelligent log analytics for NSX. Log Insight provides monitoring and troubleshooting capabilities and customizable dashboards for network virtualization, flow analysis, and alerts. The 3.3.2 version of Log Insight accepts NSX Standard/Advanced/Enterprise edition license keys issued for NSX 6.2.2 and above. This means you will have an enterprise level Log Insight license for very license of NSX.

log analytics for NSX

About the authors:

Host disconnected from vCenter and VMs showing as inaccessible

Another deep-dive troubleshooting blog today from Nathan Small (twitter account: @vSphereStorage)
 
Description from customer:
 
Host is getting disconnected from vCenter and VMs are showing as inaccessible. Only one host is affected.
 
 
Analysis:
 
A quick review of the vmkernel log shows a log spew of H:0x7 errors to numerous LUNs. Here is a short snippet where you can see how frequently they are occurring (multiple times per second):
 
# cat /var/log/vmkernel.log
 
2016-01-13T18:54:42.994Z cpu68:8260)ScsiDeviceIO: 2326: Cmd(0x412540b96e80) 0x28, CmdSN 0x8000006b from world 11725 to dev “naa.600601601b703400a4f90c3d0668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:43.027Z cpu68:8260)ScsiDeviceIO: 2326: Cmd(0x4125401b2580) 0x28, CmdSN 0x8000002e from world 11725 to dev “naa.600601601b70340064a24ada10fae211” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:43.030Z cpu68:8260)ScsiDeviceIO: 2326: Cmd(0x4125406d5380) 0x28, CmdSN 0x80000016 from world 11725 to dev “naa.600601601b7034000c70e4e610fae211” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:43.542Z cpu67:8259)ScsiDeviceIO: 2326: Cmd(0x412540748800) 0x28, CmdSN 0x80000045 from world 11725 to dev “naa.600601601b70340064a24ada10fae211” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:43.808Z cpu74:8266)ScsiDeviceIO: 2326: Cmd(0x412541229040) 0x28, CmdSN 0x8000003c from world 11725 to dev “naa.600601601b7034008e56670a11fae211” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:44.088Z cpu38:8230)ScsiDeviceIO: 2326: Cmd(0x4124c0ff4f80) 0x28, CmdSN 0x80000030 from world 11701 to dev “naa.600601601b703400220f77ab15fae211” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:44.180Z cpu74:8266)ScsiDeviceIO: 2326: Cmd(0x412540ccda80) 0x28, CmdSN 0x80000047 from world 11725 to dev “naa.600601601b70340042b582440668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:44.741Z cpu61:8253)ScsiDeviceIO: 2326: Cmd(0x412540b94480) 0x28, CmdSN 0x80000051 from world 11725 to dev “naa.600601601b70340060918f5b0668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:44.897Z cpu63:8255)ScsiDeviceIO: 2326: Cmd(0x412540ff3180) 0x28, CmdSN 0x8000007a from world 11725 to dev “naa.600601601b7034005c918f5b0668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:45.355Z cpu78:8270)ScsiDeviceIO: 2326: Cmd(0x412540f3b2c0) 0x28, CmdSN 0x80000039 from world 11725 to dev “naa.600601601b70340060918f5b0668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:45.522Z cpu70:8262)ScsiDeviceIO: 2326: Cmd(0x41254073d0c0) 0x28, CmdSN 0x8000002c from world 11725 to dev “naa.600601601b7034000e3e97350668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:45.584Z cpu71:8263)ScsiDeviceIO: 2326: Cmd(0x412541021780) 0x28, CmdSN 0x80000067 from world 11725 to dev “naa.600601601b7034000e3e97350668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:45.803Z cpu63:8255)ScsiDeviceIO: 2326: Cmd(0x412540d20480) 0x28, CmdSN 0x80000019 from world 11725 to dev “naa.600601601b703400d24fc7620668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-01-13T18:54:46.253Z cpu74:8266)ScsiDeviceIO: 2326: Cmd(0x412540b96380) 0x28, CmdSN 0x8000006f from world 11725 to dev “naa.600601601b7034005e918f5b0668e311” failed H:0x7 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
 
The Host side error (H:0x7) literally translates to Storage Initiator Error, which makes it sounds like there is something physical wrong with the card. One needs to understand that this status is sent up the stack from the HBA driver so really it is up to the those that write the driver to use this status for certain conditions. As there are no accompanying errors from the HBA driver, which in this case is a Brocade HBA, this is all we have to work with without enabling verbose logging in the driver. Verbose logging requires a reboot so this is not always an option when investigating root cause. The exception would be that the issue in ongoing so rebooting a host to capture this data is a viable option.
 
Taking a LUN as an example from ‘esxcfg-mpath -b’ output to get a view of the paths and targets:
 
# esxcfg-mpath -b
 
naa.600601601b703400b6aa124c0668e311 : DGC Fibre Channel Disk (naa.600601601b703400b6aa124c0668e311)
   vmhba0:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9a WWPN: 20:01:74:86:7a:ae:1c:9a  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:63:47:20:7a:a8
   vmhba1:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba0:C0:T1:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9a WWPN: 20:01:74:86:7a:ae:1c:9a  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:6b:47:20:7a:a8
   vmhba1:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
   vmhba2:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:32 WWPN: 20:01:74:86:7a:ae:1c:32  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:63:47:20:7a:a8
   vmhba3:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba2:C0:T1:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:32 WWPN: 20:01:74:86:7a:ae:1c:32  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:6b:47:20:7a:a8
   vmhba3:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
 
Let’s look at the adapter statistics for all HBAs. I would recommend always using localcli over esxcli when troubleshoot as esxcli requires hostd to be functioning properly:
 
# localcli storage core adapter stats get
 
vmhba0:
   Successful Commands: 844542177
   Blocks Read: 243114868277
   Blocks Written: 25821448417
  Read Operations: 395494703
   Write Operations: 405753901
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 35403
   Failed Blocks Read: 57744
   Failed Blocks Written: 16843
   Failed Read Operations: 8224
   Failed Write Operations: 16450
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0
 
vmhba1:
   Successful Commands: 502595840 <– Far less successful commands than the other adapters
   Blocks Read: 116436597821
   Blocks Written: 16509939615
   Read Operations: 216572537
   Write Operations: 245276523
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 10942696
   Failed Blocks Read: 12055379188 <– 12 billion failed blocks read! Other adapters are all less than 60,000
   Failed Blocks Written: 933809
   Failed Read Operations: 10895926
   Failed Write Operations: 25645
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0
 
vmhba2:
   Successful Commands: 845976973
   Blocks Read: 244034940187
   Blocks Written: 26063852941
   Read Operations: 397564994
   Write Operations: 407538414
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 40468
   Failed Blocks Read: 44157
   Failed Blocks Written: 18676
   Failed Read Operations: 5506
   Failed Write Operations: 12152
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0
 
vmhba3:
   Successful Commands: 866718515
   Blocks Read: 249837164491
   Blocks Written: 26492209531
   Read Operations: 406367844
   Write Operations: 416901703
   Reserve Operations: 0
   Reservation Conflicts: 0
   Failed Commands: 37723
   Failed Blocks Read: 23191
   Failed Blocks Written: 139380
   Failed Read Operations: 7372
   Failed Write Operations: 14878
   Failed Reserve Operations: 0
   Total Splits: 0
   PAE Commands: 0
 
 
Let’s see how often the vmkernel.log reports messages for that HBA:
 
# cat vmkernel.log |grep vmhba0|wc -l
112
 
# cat vmkernel.log |grep vmhba1|wc -l
8474 <– over 8000 times this HBA is mentioned! This doesn’t mean they are all errors, of course, but based on the log spew we know is already occurring it means it likely is
 
# cat vmkernel.log |grep vmhba2|wc -l
222
 
# cat vmkernel.log |grep vmhba3|wc -l
335
 
Now let’s take a look at the zoning to see if multiple adapters are zoned to the exact same array targets (WWPN) in attempt to determine if the issue is possibly array side or HBA side:
 
# esxcfg-mpath -b
 
naa.600601601b703400b6aa124c0668e311 : DGC Fibre Channel Disk (naa.600601601b703400b6aa124c0668e311)
   vmhba0:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9a WWPN: 20:01:74:86:7a:ae:1c:9a  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:63:47:20:7a:a8
   vmhba1:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba0:C0:T1:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9a WWPN: 20:01:74:86:7a:ae:1c:9a  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:6b:47:20:7a:a8
   vmhba1:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
   vmhba2:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:32 WWPN: 20:01:74:86:7a:ae:1c:32  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:63:47:20:7a:a8
   vmhba3:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba2:C0:T1:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:32 WWPN: 20:01:74:86:7a:ae:1c:32  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:6b:47:20:7a:a8
   vmhba3:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
 
Let’s isolate the HBAs so they are easier to visually compare the WWPN of the array targets:
 
vmhba1:
 
   vmhba1:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba1:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:9c WWPN: 20:01:74:86:7a:ae:1c:9c  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
 
vmhba3:
 
   vmhba3:C0:T3:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:60:47:24:7a:a8
   vmhba3:C0:T2:L20 LUN:20 state:active fc Adapter: WWNN: 20:00:74:86:7a:ae:1c:34 WWPN: 20:01:74:86:7a:ae:1c:34  Target: WWNN: 50:06:01:60:c7:20:7a:a8 WWPN: 50:06:01:68:47:24:7a:a8
 
vmhba1 and vmhba3 are zoned to the exact same array ports yet only vmhba1 is experiencing communication issues/errors.
 
 
Let’s look at the driver information under /proc/scsi/bfa/ by viewing (cat) the node information:
 
Chip Revision: Rev-E
Manufacturer: Brocade
Model Description: Brocade-1741
Instance Num: 0
Serial Num: xxxxxxxxx32
Firmware Version: 3.2.3.2
Hardware Version: Rev-E
Bios Version: 3.2.3.2
Optrom Version: 3.2.3.2
Port Count: 2
WWNN: 20:00:74:86:7a:ae:1c:9a
WWPN: 20:01:74:86:7a:ae:1c:9a
Instance num: 0
Target ID: 0 WWPN: 50:06:01:6b:47:20:7b:04
Target ID: 1 WWPN: 50:06:01:6b:47:20:7a:a8
Target ID: 2 WWPN: 50:06:01:63:47:20:7b:04
Target ID: 3 WWPN: 50:06:01:63:47:20:7a:a8
 
Chip Revision: Rev-E
Manufacturer: Brocade
Model Description: Brocade-1741
Instance Num: 1
Serial Num: xxxxxxxxx32
Firmware Version: 3.2.3.2
Hardware Version: Rev-E
Bios Version: 3.2.3.2
Optrom Version: 3.2.3.2
Port Count: 2
WWNN: 20:00:74:86:7a:ae:1c:9c
WWPN: 20:01:74:86:7a:ae:1c:9c
Instance num: 1
Target ID: 0 WWPN: 50:06:01:60:47:24:7b:04
Target ID: 1 WWPN: 50:06:01:68:47:24:7b:04
Target ID: 3 WWPN: 50:06:01:60:47:24:7a:a8
Target ID: 2 WWPN: 50:06:01:68:47:24:7a:a8
 
Chip Revision: Rev-E
Manufacturer: Brocade
Model Description: Brocade-1741
Instance Num: 2
Serial Num: xxxxxxxxx2E
Firmware Version: 3.2.3.2
Hardware Version: Rev-E
Bios Version: 3.2.3.2
Optrom Version: 3.2.3.2
Port Count: 2
WWNN: 20:00:74:86:7a:ae:1c:32
WWPN: 20:01:74:86:7a:ae:1c:32
Instance num: 2
Target ID: 0 WWPN: 50:06:01:6b:47:20:7b:04
Target ID: 1 WWPN: 50:06:01:6b:47:20:7a:a8
Target ID: 2 WWPN: 50:06:01:63:47:20:7b:04
Target ID: 3 WWPN: 50:06:01:63:47:20:7a:a8
 
Chip Revision: Rev-E
Manufacturer: Brocade
Model Description: Brocade-1741
Instance Num: 3
Serial Num: xxxxxxxxx2E
Firmware Version: 3.2.3.2
Hardware Version: Rev-E
Bios Version: 3.2.3.2
Optrom Version: 3.2.3.2
Port Count: 2
WWNN: 20:00:74:86:7a:ae:1c:34
WWPN: 20:01:74:86:7a:ae:1c:34
Instance num: 3
Target ID: 0 WWPN: 50:06:01:60:47:24:7b:04
Target ID: 1 WWPN: 50:06:01:68:47:24:7b:04
Target ID: 2 WWPN: 50:06:01:68:47:24:7a:a8
Target ID: 3 WWPN: 50:06:01:60:47:24:7a:a8
 
So all HBAs are the same firmware, which is important from a observed consistency perspective. Had the firmware versions been different then there might be something to go on, or at least verify whether there are issues with that firmware level. Obviously they are using the same driver as well since only one is loaded in the kernel.
 
We can see not only by the shared serial number above but also by the lspci output that these are 2 port physical cards:
 
# lspci
 
000:007:00.0 Serial bus controller: Brocade Communications Systems, Inc. Brocade-1010/1020/1007/1741 [vmhba0]
000:007:00.1 Serial bus controller: Brocade Communications Systems, Inc. Brocade-1010/1020/1007/1741 [vmhba1]
000:009:00.0 Serial bus controller: Brocade Communications Systems, Inc. Brocade-1010/1020/1007/1741 [vmhba2]
000:009:00.1 Serial bus controller: Brocade Communications Systems, Inc. Brocade-1010/1020/1007/1741 [vmhba3]
 
The first set of numbers are read as Domain:Bus:Slot.Function so vmhba0 and vmhba1 are both on Domain 0, Bus 7, Slot 0, amd function 0 and 1 respectively, which means it is a dual port HBA.
 
So vmhba0 and vmhba1 are the same physical card yet only vmhba1 is showing errors. The HBA chips themselves on a dual port HBA are mostly independent of each other so at least this means there isn’t a problem with the board or circuitry they both share. I say mostly since the physical ports are independent of each other as well as the HBA chip however they do share the same physical board and connection on the motherboard.
 
This host is running EMC PowerPath VE so we know that in general the I/O loads is evenly distributed across all HBAs and paths evenly. I say in general as PowerPath VE is intelligent enough to use paths that exhibit more errors than other paths less frequently, as well as paths that are more latent.
 
I believe we may be looking at either a cable issue (loose, faulty, or bad GBIC) between vmhba1 and the switch or the switch port itself that vmhba1 is connected to. Here is why:
 
1. vmhba1 is seeing thousands upon thousands of errors while the other HBAs are very quiet
2. vmhba1 and vmhba3 are zoned to the exact same targets yet only vmhba1 is seeing errors
3. vmhba0 and vmhba1 are the same physical card yet only vmhba 1 is seeing errors
 
My recommendation would be to check the physical switch port error counters and possibly replace the cable to see if the errors subside. It is standard practice to reset the switch counters and monitor to ensure errors are still happening so may be needed to do that to validate that the CRC errors or other fabric errors are still occurring.
 
Cheers,
Nathan (twitter account: @vSphereStorage)

New KB articles published for week ending 8/29/15

VMware App Volumes
Using single ESXi host as the Machine Manager in VMware App Volumes generates error: “appvolumes rejected all 2 volumes because they are invalid or have no accessible volume file” (2127638)
Date Published: 8/26/2015

VMware ESXi
Accessing the same storage through different transport protocols, such as iSCSI and Fibre Channel is unsupported (2123036)
Date Published: 8/28/2015
VMware mouse and video drivers not working after upgrading to SLE12. (2120590)
Date Published: 8/28/2015
vMotion fails at 15% with the error: A specified parameter was not correct (2127211)
Date Published: 8/26/2015
Fixing the primary GPT table using the secondary (backup) GPT table (2121741)
Date Published: 8/27/2015

VMware Fusion
Upgrading from Fusion 7 to Fusion 8 (2126734)
Date Published: 8/25/2015
VMware Fusion 8.x, VMware Workstation 12.x Pro and Workstation 12.x Player FAQs (2129663)
Date Published: 8/25/2015
Downgrading VMware Fusion 8.x to 7.x (2128615)
Date Published: 8/25/2015
Installing Windows 10 as a guest operating system in VMware Fusion (2128765)
Date Published: 8/25/2015
Supported host operating systems for VMware Fusion 8.x and VMware Fusion Pro 8.x (2129865)
Date Published: 8/26/2015

VMware Fusion Pro
VMware Fusion 8.0 sales, licensing and compatibility FAQs (2126715)
Date Published: 8/25/2015

VMware Horizon DaaS
Collecting diagnostic information from VMware Horizon DaaS Appliances (2121318)
Date Published: 8/26/2015

VMware NSX for vSphere
Creating or upgrading VMware NSX for vSphere 6.x Edges fails after adding IPv6 address under NSX Management interface (2127561)
Date Published: 8/27/2015

VMware vCenter Infrastructure Navigator
Enabling SSH and permitting root login in VMware vRealize Infrastructure Navigator (2126791)
Date Published: 8/27/2015

VMware vCenter Server
Get-TagCategory PowerCLI command fails with the error: Cannot find a service registration for vCenter service with host name (2127028)
Date Published: 8/26/2015
Determining replication agreements and status with the Platform Services Controller 6.0 (2127057)
Date Published: 8/26/2015
Unable to open the VMRC console in the VMware vCenter Web Client using Users with Console Interaction privileges (2128080)
Date Published: 8/26/2015
Reconnecting a virtual machine that has been vMotioned with a network card in a disconnected state (Connected checkbox in the virtual machine’s Edit Settings tab is unchecked) fails (2120443)
Date Published: 8/27/2015
Changing the vCenter Server 6.0 service recovery response (2127930)
Date Published: 8/26/2015

VMware vCenter Server Appliance
Upgrading VMware vCenter Server Appliance 5.5 to 6.0 gives warning message: Newly deployed appliance FQDN will be set to ip_address (2127750)
Date Published: 8/26/2015
VMware Virtual Center Server Appliance host name and DNS configuration revert after a reboot. (2122383)
Date Published: 8/28/2015

VMware vCloud Networking and Security
Attempting to register VMware vShield Manager to the vCenter Server 6.0 SSO Lookup Service fails with the error: SSL certificate of the STS service cannot be verified (2121696)
Date Published: 8/27/2015

VMware Virtual SAN
VMware virtual SAN storage provider fails during resyncwith the error: com.vmware.vim.binding.sms.fault.ServiceNotInitialized (2129032)
Date Published: 8/26/2015
Considerations when using both VSAN and non-VSAN disks with the same storage controller (2129050)
Date Published: 8/24/2015
VSAN handling of devices detected as snapshots (2129058)
Date Published: 8/25/2015

VMware vRealize Automation
VMware vRealize Orchestrator endpoint data collection fails (2123455)
Date Published: 8/28/2015

VMware vRealize Log Insight
VMware vRealize Log Insight deployment with reused IP address fails to initialize (2126692)
Date Published: 8/28/2015

VMware vRealize Operations Manager
Top-N widget does not work properly when a metric with on/off or 0/1 values is chosen In VMware vRealize Operations Manager 6.0.x (2123634)
Date Published: 8/28/2015
vRealize Operations Manager: The Compliance dashboard shows non-compliance for old virtual machines (2128501)
Date Published: 8/24/2015
Upgrade to the Management Pack for Storage Devices 6.0.2 (GA) is not supported from previous releases of the Management Pack for Storage Devices (2129378)
Date Published: 8/24/2015
How to delete Launch In Context menu items after deleting Log Insight server (2126679)
Date Published: 8/28/2015

VMware Workstation
Upgrading from VMware Workstation 11 to VMware Workstation Pro 12 (2126729)
Date Published: 8/25/2015
Installing Windows 10 as a guest operating system in VMware Workstation Pro 12.x (2128790)
Date Published: 8/25/2015
Supported host operating systems for VMware Workstation Pro 12.x (2129859)
Date Published: 8/26/2015
Downgrading from VMware Workstation Pro 12 to VMware Workstation 11 (2128613)
Date Published: 8/25/2015
USB 3.0 Support for Windows 7 virtual machine in VMware Workstation and VMware Fusion (2128105)
Date Published: 8/26/2015
VMware Workstation on a Windows host fails to properly display the desktop of a virtual machine and logs graphics-related errors (2128098)
Date Published: 8/24/2015

VMware Workstation Player
Upgrading from Player 7 to Workstation Player 12 (2126733)
Date Published: 8/25/2015
Licensing VMware Fusion 8.x, VMware Workstation 12.x Pro and Workstation 12.x Player (2126889)
Date Published: 8/25/2015
Installing Windows 10 as a guest operating system in VMware Workstation 12.x Player (2128797)
Date Published: 8/25/2015
VMware Workstation Player 12.x sales, licensing, and compatibility FAQs (2126732)
Date Published: 8/25/2015
Technology Guarantee Program (TGP) and Subscription Fulfillment for VMware Workstation Pro and Workstation Player (2129671)
Date Published: 8/25/2015

VMware Workstation Pro
VMware Workstation 12.x Pro sales, licensing, and compatibility FAQs (2126721)
Date Published: 8/25/2015
Recording with TechSmith Camtasia Recorder on certain Windows guests with a non-standard resolution might cause Camtasia Recorder to crash (2129526)
Date Published: 8/24/2015

New KB articles published for week ending 8/15/15

VMware ESX
Understanding the difference between Not Responding and Disconnected ESXi hosts in VMware vCenter Server (2121031)
Date Published: 8/13/2015

VMware ESXi
Viewing properties of a storage adapter from the VMware vSphere Client fails with the error: An internal error occurred in the vSphere Client (2126657)
Date Published: 8/10/2015
Deploying a virtual machine from OVF fails with the error: Failed to deploy OVF package: Incompatible device backing specified for device ‘9’ (2128084)
Date Published: 8/13/2015
Virtual machines might experience slow performance due to high input/output processing (2118895)
Date Published: 8/14/2015
Large Windows 10 Virtual Machines Are Slow to Boot (2127782)
Date Published: 8/13/2015

VMware Fusion
Known issues with Windows 10 on VMware Fusion and VMware Workstation (2125739)
Date Published: 8/11/2015

VMware Horizon Flex
Uninstalling VMware Tools does not remove the VMware RVM Setup Service (2123935)
Date Published: 8/14/2015

VMware Horizon View
German Umlaute characters do not input when using Firefox 26 or later and HTML Access (2119698)
Date Published: 8/12/2015
Attempting to access VMware View desktops or apps using HTML5 BLAST from Internet Explorer causes error: HTTP Error 404 Web Page Not Found (2127422)
Date Published: 8/14/2015
View 5.3.5 Transfer Server requires Microsoft Visual C++ 2010 SP1 Redistributable Package (2127795)
Date Published: 8/10/2015

VMware NSX for vSphere
Understanding and troubleshooting VMware NSX for vSphere 6.x Distributed Firewall (DFW) (2125437)
Date Published: 8/12/2015
Troubleshooting Network and Security Service Insertion in VMware NSX for vSphere (2125482)
Date Published: 8/11/2015
Understanding and troubleshooting VMware vSphere ESX Agent Manager (EAM) with NSX for vSphere 6.x (2122392)
Date Published: 8/10/2015
Understanding and troubleshooting IPSec VPN in NSX for vSphere 6.x (2123580)
Date Published: 8/14/2015
Understanding and troubleshooting SSL VPN-Plus for VMware NSX for vSphere 6.x (2126671)
Date Published: 8/10/2015
IGMP support in NSX for vSphere (2120827)
Date Published: 8/13/2015
Installing Guest Introspection on a host that is offline shows Installation Status as Succeeded and a Status as Unknown (2126412)
Date Published: 8/14/2015
Oracle connections times out when forwarded through the VMware NSX for vSphere 6.1.x Edge (2126674)
Date Published: 8/14/2015
Supported methods for moving VMware NSX for vSphere 6.x prepared hosts across clusters (2127366)
Date Published: 8/14/2015

VMware vCenter Server
Installing or Upgrading to vCenter Server 6.0 fails with the error: The following command could not initialize the Content Library Server database (2128206)
Date Published: 8/14/2015
Installing the VMware vCenter Server6.0 fails with the vminst.log error: MSI result of install of “C:\vCenter-Server\Packages\vcsservicemanager.msi” may have failed : 1603 (2127519)
Date Published: 8/11/2015

VMware vCloud Air
The Reverse Stretch Deploy Script doesn’t work for VMware vCloud Air users who do not have the Organization Administrator role in vCloud Director (2127081)
Date Published: 8/12/2015
Adding purchased IP addresses in VMware vCloud Air when they do not display automatically (2127205)
Date Published: 8/12/2015

VMware vCloud Automation Center for Server
Delegating permissions across domains using Domain Local groups might not work as expected with vCenter Server or vRealize Automation (2128100)
Date Published: 8/14/2015

VMware vCloud Government Service provided by Carpathia Dedicated Cloud
Cannot connect to the vCloud Government Services (vCGS) API or to vCGS using vCloud Connector (vCC) (2117454)
Date Published: 8/14/2015

VMware vCloud Networking and Security
VMware vCloud Networking and Security Manager support of vShield Endpoint (2128156)
Date Published: 8/14/2015

VMware View Manager
Provisioning of VMware View desktops fails with Error : View Composer Agent Initialization Error (16):Failed to Activate Software License (waited xxxx seconds) (2125106)
Date Published: 8/12/2015

VMware vRealize Automation
AWS Government cloud and C2S cloud support in VMware vRealize Automation (2126742)
Date Published: 8/13/2015
Deploying a virtual machine using a clone workflow fails in vRealize Automation 6.x (2127627)
Date Published: 8/10/2015

VMware vRealize Operations Manager
Cancelling alerts via the UI does not result in the selected alerts being removed in VMware vRealize Operations Manager 6.0.x (2123921)
Date Published: 8/13/2015
In VMware vRealize Operations Manager 6.0.x the Dashboard Reorder/Autoswitch list only shows 25 dashboards (2127975)
Date Published: 8/13/2015
Objects are missing from the VMware vRealize Operations Manager 6.x inventory when logged in as a vSphere vCenter Server authenticated user (2128010)
Date Published: 8/13/2015
Widgets do not display data and cannot be edited on dashboards imported from a different VMware vRealize Operations Manager installation (2122016)
Date Published: 8/12/2015
vRealize Operations Manager 6.0.0 deployment fails on vCloud (2127663)
Date Published: 8/13/2015
A blob error is generated when logging in to VMware vRealize Operations Manager or when viewing dashboards (2128077)
Date Published: 8/13/2015

VMware vSphere Replication
Deploying vSphere replication 6.0 fails while registering with lookup service (2119343)
Date Published: 8/14/2015

VMware Workstation (for Linux)
Creating symbolic links in a Windows guest operating system on a Linux host does not list files under subdirectories (2123633)
Date Published: 8/14/2015

New KB articles published for week ending 8/8/15

VMware App Volumes
A large number of attached applications causes truncated environment variables when using VMware App Volumes on Windows Vista, 7, 2008 or 2008R2 (2123929)
Date Published: 8/8/2015

VMware ESXi
Red Hat Enterprise Linux 6.5 panics after installing an Operating Specific Package (OSP) version of VMware Tools (2125636)
Date Published: 8/5/2015
A Windows virtual machine using the VMXNET3 virtual network adapter that is assigned a multicast MAC address is not recognized (2127248)
Date Published: 8/7/2015
ESXi host loses storage access with the vmkernel.log errors: SCSILinuxAbortCommands:1843: Failed, Driver hpsa, for vmhbax (2127502)
Date Published: 8/6/2015
Adding an ESXi host with a similar CPU to the current EVC baseline (2127212)
Date Published: 8/3/2015
Adding an ESXi host with an older CPU than the current EVC baseline (2127208)
Date Published: 8/3/2015
After upgrading HBA card driver , storage adapter names might change for some of the FC vmhbas (2127274)
Date Published: 8/5/2015

VMware Mirage
VMware Mirage 5.x Web Management Console installs but causes application pool to stop (2126277)
Date Published: 8/9/2015
Unable to add second management server to the mirage cluster with SQL Always On Feature (2127048)
Date Published: 8/9/2015

VMware NSX for vSphere
Troubleshooting VXLAN Installation in NSX for vSphere (2124894)
Date Published: 8/7/2015
Adding more than 255 characters in the source or destination field for an NSX Distributed Firewall L3 rule fails in VMware NSx for vSphere 6.1.x (2127649)
Date Published: 8/6/2015
VMware NSX for vSphere 6.x Controllers disconnects intermittently (2127655)
Date Published: 8/6/2015
Troubleshooting NSX for vSphere Edge Load Balancer (2122708)
Date Published: 8/7/2015
Understanding and troubleshooting High Availability (HA) on the NSX for vSphere 6.x Edge (2126560)
Date Published: 8/4/2015

VMware vCenter Operations Manager (vApp)
Custom Reports no longer function after upgrading VMware vCenter Operations Manager to version 5.8.5 (2127573)
Date Published: 8/7/2015

VMware vCenter Server
Enabling debug logging for Inventory Service in the vCenter Server 6.0 (2125609)
Date Published: 8/4/2015
Determining the ownership of VMware vCenter Server 6.0 Components within the vSphere Domain (2126660)
Date Published: 8/7/2015
Enabling and managing EVC on ESXi clusters in vCenter Server 5.x and 6.0 (2124932)
Date Published: 8/3/2015
Opening a console to a virtual machine in the VMware vSphere Web Client fails with the error: Failed to match the SSL thumbprint for “vCenterFQDN”. Make sure that all security patches have been installed on the server and then retry the connection. (2124868)
Date Published: 8/5/2015
Improving transfer speed of task with library items (2112692)
Date Published: 8/3/2015
Triaging a vCenter Server 6.0 installation or upgrade failure with 1603 errors (2127284)
Date Published: 8/5/2015

VMware vCenter Server Appliance
Upgrading vCenter Server Appliance 5.5 to 6.0 that has been changed to a static host name fails (2127059)
Date Published: 8/3/2015
Unable to add Active Directory Users or Groups to vCenter Server Appliance or vRealize Automation permissions (2127213)
Date Published: 8/4/2015

VMware vCenter Update Manager
Unattended Installation of VMware vCenter Update Manager 6.0 (2126111)
Date Published: 8/6/2015

VMware vCloud Air
VMware Mirage with SSL in a VMware vCloud Air environment (2126255)
Date Published: 8/4/2015
How to build the vCloud Connector Node address for a Virtual Private Cloud OnDemand or Disaster Recovery to the Cloud environment (2126268)
Date Published: 8/5/2015

VMware vCloud Automation Center
Troubleshooting Model Manager Data installation failures in VMware vRealize Automation (2091551)
Date Published: 8/5/2015

VMware vFabric Hyperic Agent
Manage core dump files from VMware vRealize Hyperic Agent (2125333)
Date Published: 8/5/2015

VMware View
Unable to use USB redirection after renaming the hostname of view desktop (2127445)
Date Published: 8/8/2015

VMware vRealize Operations Manager
The VMware vRealize Operations Manager Health Weather Map widget cannot be interacted with when the number of child resources exceeds 1000 (2116424)
Date Published: 8/4/2015
Change the IP address on a VMware vRealize Operations Manager single-node deployment (2126066)
Date Published: 8/4/2015
In VMware vRealize Operations Manager the output/values are wrong when using the Projects Capacity Planning with one scenario with multiple Hosts (2115478)
Date Published: 8/4/2015

VMware vSphere CLI
add-esxsoftwaredepot Cmdlet does not support umlaut characters in the directory path (2126109)
Date Published: 8/6/2015

VMware vSphere Data Protection
Configuring vSphere Data Protection 6.0.x replication fails with an authentication error when only port 29000 is open (2127431)
Date Published: 8/5/2015

VMware vSphere Replication
Configuring the NTP server in vSphere Replication 6.0 fails with the error: Error: Cannot update the dynamic configuration (2126965)
Date Published: 8/5/2015

VMware Workspace Portal
VMware Workspace 2.x unable to sync with Thinapp Repository (2126273)
Date Published: 8/4/2015

New KB articles published for week ending 7/25/15

VMware App Volumes
VMware App Volumes log analysis tools and log analysis examples (2126475)
Date Published: 7/24/2015
VMware App Volumes log files, log analysis tools, and log analysis examples (2125908)
Date Published: 7/23/2015

VMware ESXi
Disabling VAAI ATS heartbeat using Host Profiles and Auto-Deploy (2123928)
Date Published: 7/22/2015
While performing a test failover in VMware Site Recovery Manager 5.8 virtual machine Power On task fails with Error : Operation timed out: 1020 seconds (2125902)
Date Published: 7/20/2015
Emulex lpfc native driver stuck on abort handling with timeout (2125904)
Date Published: 7/20/2015
Servers/Systems used with Riverbed SteelFusion Edge Hyper converged Solution (Partner Support) (2125627)
Date Published: 7/20/2015

VMware EVO:RAIL
Locating VMware EVO:RAIL Release 1.2.1 Release Notes and downloads (2125254)
Date Published: 7/20/2015

VMware Horizon DaaS
Horizon DaaS/Air – Convert to Gold Pattern process fails with a timeout error for Windows OS (2126179)
Date Published: 7/21/2015

VMware vCenter Chargeback Manager
CVE-2015-0204 : FREAK Attack in Chargeback Manager 2.6 and 2.7 (2123548)
Date Published: 7/23/2015

VMware vCenter Server
After replacing the VMware vCenter Server certificates in VMware vSphere 6.0, the VMware vSphere Auto Deploy solution user fails to log in (2123631)
Date Published: 7/20/2015
Configuring email alert for vCenter Server alarm (2123925)
Date Published: 7/21/2015
Using foreign key (FK) constraint checker script to detect potential cause of orphan data existence in the vCenter Server SQL database causing the VMware Virtual Center Server to fail to start (2124695)
Date Published: 7/23/2015
vCenter Server 6.0 fails to display the node count within the environment for a user (2124899)
Date Published: 7/23/2015
Installing or Upgrading to vCenter Server 6.0 fails with the error: Unable to get port number for mssql provider (2125492)
Date Published: 7/20/2015
Local users created in VMware vCenter Single Sign-On 6.0 fail to log in after modifying the maximum lifetime value for password expiration (2125495)
Date Published: 7/22/2015
After updating Mozilla Firefox to Version 39 or later, vSphere Web Client Integration Plugin 6.0 no longer functions (2125623)
Date Published: 7/23/2015
Deploying vCenter Single Sign-On 5.5 in a HA Configuration (2091683)
Date Published: 7/23/2015
Users with restricted permissions are unable to view advanced performance data charts in VMware vCenter Server 6.0 (2123566)
Date Published: 7/21/2015
Upgrading a vCenter Server protected by vCenter Server Heartbeat to vCenter Server 6.0 (2123927)
Date Published: 7/20/2015
Accessing a virtual machine console using the vSphere Web Client in vCenter Server 6.0 fails (2126280)
Date Published: 7/24/2015

VMware vCenter Server Appliance
Reviewing and Managing Local and Global Permissions in the vCenter Server 6.0 (2123931)
Date Published: 7/21/2015
Attempting to change the IP address of the VMware vCenter Server Appliance 6.0 fails with the error: IPv4 configuration for nic0 of this node cannot be edited post deployment (2124422)
Date Published: 7/22/2015
The vCenter Server Appliance 6.0 HTML Installer gives the error: Failed to read from file: vmware-vcsa (2125737)
Date Published: 7/23/2015
Increasing the disk space for the vCenter Server Appliance in vSphere 6.0 (2126276)
Date Published: 7/24/2015
Upgrading the vCenter Server Appliance to 6.0 fails with the error: Error during upgrade: Guest operation Initiate File Transfer From Guest (2125629)
Date Published: 7/20/2015

VMware vCenter Site Recovery Manager
IP customization silently fails in Windows 7 or Windows 2008 R2 guests when running a test failover in VMware Site Recovery Manager (2125178)
Date Published: 7/21/2015

VMware vCloud Air
Copying a virtual machine to the VMware On-Demand Cloud using VMware vCloud Connector fails when the disk provisioning is set to Thick Provision Eager Zeroed and the UDT protocol is employed (2124602)
Date Published: 7/21/2015

VMware vCloud Automation Center
Provisioning multiplevirtual machines in the VMware vRealize Automation with external workflows fails with Timeout on signal errors (2116936)
Date Published: 7/22/2015
VMware vRealize Orchestrator and NSX for vSphere 6.x plug-in compatibility with vRealize Automation 6.1.x (2126258)
Date Published: 7/23/2015
VMware vRealize Automation vCenter Server Proxy Agent stops without errors indicated (2126265)
Date Published: 7/23/2015

VMware vCloud Director
Calling the SSOConnection SDK reports the exception: Client received SOAP Fault from server: The time now does not fall in the request lifetime interval extended with clock tolerance of 600000 ms (2125193)
Date Published: 7/21/2015
After deploying a vApp from VMware vCloud Director guest operating system customization does not initiate (2126439)
Date Published: 7/24/2015

VMware vCloud Director for Service Provider
The VMware vCloud Director 5.6.4 service fails to start after upgrade with the error: Could not instantiate class [com.vmware.vcloud.logging.CustomPatternLayout] (2114462)
Date Published: 7/20/2015

VMware vRealize Automation
Renaming a VMware vSphere Cluster used in a VMware vRealize Automation environment (2114494)
Date Published: 7/20/2015

VMware vRealize Operations Manager
Restarting VMware vRealize Operations Manager 6.0.x fails with a Waiting for Analytics message in the Admin UI (2114663)
Date Published: 7/23/2015
VMware vRealize Operations Manager 6.0 displays the critical alert: FSDB file corrupted for resourceInternalId (2126139)
Date Published: 7/23/2015

VMware vSphere Data Protection
Asia/Rangoon timezone is not available in VMware vSphere Data Protection 6.0 (2125513)
Date Published: 7/23/2015

New KB articles published for week ending 7/18/15

My VMware
My VMware Best Practices for License Management (2124980)
Date Published: 7/15/2015
Virtual Disk Development Kit
Backing up a virtual machine using Virtual Disk Development Kit (VDDK) 6.0 crashes while attempting to open a disk on the Windows Proxy (2125497)
Date Published: 7/16/2015
VMware App Volumes
VMware App Volumes deployment requirements (2124876)
Date Published: 7/13/2015
Troubleshooting VMware App Volumes through the App Volumes Manager console (2125263)
Date Published: 7/14/2015
Troubleshooting VMware App Volumes using the SQL database (2125753)
Date Published: 7/17/2015
Confirming VMware App Volumes environment information from registry entries (2125612)
Date Published: 7/16/2015
VMware ESXi
After changing the Logical Unit Number state from Read-Only to Read-Write virtual machine RDMs remain in Read-Only/Write Protected mode (2118333)
Date Published: 7/14/2015
Upgrade Windows Server 2003 in a VM to a later OS (2120918)
Date Published: 7/15/2015
esxtop csv export is missing vscsi performance data (2123383)
Date Published: 7/13/2015
vMotion fails with Error: Failed to attach filter ‘XXX’ to scsi (2125133)
Date Published: 7/13/2015
VMware NSX for vSphere
NSX for vSphere VXLAN logical switch connectivity tests fail with the error: The VMKPing fault, [ setsockopt() LINUX_SOCKET_SO_NETSTACK failed (Function not implemented)] (2124360)
Date Published: 7/14/2015
Distributed Firewall (DFW) rules fails to process traffic even after successfully publishing the rules in VMware NSX for vSphere 6.x (2125901)
Date Published: 7/17/2015
VMware vCenter Operations Manager (vApp)
The Reports Tabs are missing in VMware vCenter Operations Manager 5.x (2056308)
Date Published: 7/17/2015
VMware vCenter Server
After Installing or Upgrading to vCenter Server 6.0, logging into the vSphere Web Client for all users reports: You do not have permissions to view this object or this object does not exist (2125229)
Date Published: 7/13/2015
VMware vCenter Server Appliance
Creating and managing local user accounts on vCenter Server Appliance (2045805)
Date Published: 7/17/2015
VMware vCloud Automation Center for Server
Disk numbers of additional volumes are not reflected in notification email in VMware vRealize Automation (2114225)
Date Published: 7/14/2015
VMware vRealize Hyperic
Collecting diagnostic information for VMware vRealize Hyperic vApp (2125120)
Date Published: 7/13/2015
VMware vRealize Operations Manager
In VMware vRealize Operations Manager 6.0.x the classic dashboard does not have a resource assigned to it out of the box (2124893)
Date Published: 7/13/2015
Unable to remove builtin dashboards for Diagnose, Recommendation, and Self Health In VMware vRealize Operations Manager 6.0.x (2125117)
Date Published: 7/13/2015
In VMware vRealize Operations Manager 6.0.x widgets using a custom Metric Configuration file have no content or a metric graph that states No Data (2125325)
Date Published: 7/14/2015
How to enable or disable Dynamic Threshold calculations for individual objects or groups of objects (2125615)
Date Published: 7/17/2015
In VMware vRealize Operations Manager 6.0.1 the alert Distributed Switch configuration is out of sync does not clear (2125778)
Date Published: 7/17/2015
Accessing the Product UI with a node’s IP or FQDN redirects to the admin UI in vRealize Operations Manager 6.x (2125793)
Date Published: 7/17/2015
Change the vRealize Operations Manager node address format (2124922)
Date Published: 7/17/2015
Increasing the shell timeout value in the VMware vRealize Operations Manager 6.x virtual appliance (2125330)
Date Published: 7/14/2015
In VMware vRealize Operations Manager 6.0.2 the Host CPU Capacity Analysis view does not display an accurate value for an ESXi host’s CPU capacity (2125637)
Date Published: 7/17/2015