posted

0 Comments

I recently learnt about a built-in safety mechanism on the vSphere Storage Appliance when too many reboots of the host takes place. A VSA server will enter maintenance mode silently if it is rebooted 3 times in 15 minutes. If a member is in maintenance mode, then it does not join the cluster. If the cluster is in maintenance mode, then it does not provide storage. So if 2 VSA nodes end up in maintenance mode, then the cluster will go offline.

Right now, the only way to check this has occurred on the VSA is via the CLI. The WSCLI utility is installed on the vCenter server managing the VSA cluster. The easiest way to check is to use the getSvaServerInfo option to the WSCLI as follows:

cd "C:Program FilesVMwareInfrastructureVSA Managertools"

"C:Program FilesVMwareInfrastructurejrebinjava.exe" -jar WSCLI.jar <VSA IP Address> getSvaServerInfo

This should return an output similar to the following:

    id = xxxxxx-xxxx-xxxx-xxxxxxxxx
    name = localhost
    maintenance mode = true
    Domain name = localdom
    Storage Cluster ID = yyyyyyyy-yyyy-yyyy-yyyyyyyyyy
    No DNS server
    Internal interface = 192.168.4.1/24
    Management interface = A.B.C.D/22
    Gateway = A.B.C.253
    Storage pool 0:
        ID = 00000000-0000-0000-0000-000000000000
        Total storage = 19595264KB
        Free storage = 0KB
        Used storage = 19595264KB
    Free storage = 0KB
    Total storage = 19595264KB
    Used storage = 19595264KB

If maintenance mode is set to true, you can use the exitMaintenanceMode to take the host out of this state.

"C:Program FilesVMwareInfrastructurejrebinjava.exe" -jar WSCLI.jar <VSA IP Address> exitMaintenanceMode

 After typing this command, wait a minute or two before doing another getSvaServerInfo to verify that the member has indeed exited maintenance mode (maintenance mode = false). At this point, the datastores should now start syncing.

Of course, there may a number of other reasons why your cluster may be offline or a host entered maintenance mode. This post is to highlight the fact that if there are 3 successive reboots in 15 minutes, then the silent maintenance mode state is entered, and these WSCLI commands can get your VSA out of this state.

If in doubt, always reach out to your nearest VMware support representative.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: Twitter @VMwareStorage

About the Author

Cormac Hogan

Cormac Hogan is a Senior Staff Engineer in the Office of the CTO in the Storage and Availability Business Unit (SABU) at VMware. He has been with VMware since April 2005 and has previously held roles in VMware’s Technical Marketing and Technical Support organizations. He has written a number of storage related white papers and have given numerous presentations on storage best practices and vSphere storage features. He is also the co-author of the “Essential Virtual SAN” book published by VMware Press.