This is an interesting one for those of you considering using Storage I/O Control (SIOC) on NFS datastores. We’ve come across an issue a couple of times whereby SIOC fails to enable on an NFS datastore due to a permissions issue. The following messages are seen in the SIOC logs on the ESXi hosts sharing the datastore:

2013-06-10T11:53:08.147Z: <test, 0> Error in opening stat file for device: test.Ignoring this device.
2013-06-10T11:53:12.159Z: stat file /vmfs/volumes//test/.iormstats.sf already exists.
2013-06-10T11:53:12.160Z: open /vmfs/volumes//test/.iorm.sf/slotsfile(0x10000042, 0x0) failed: Permission denied
2013-06-10T11:53:12.160Z: <test, 0> Giving UP Permission denied Error -1 opening SLOT file /vmfs/volumes//test/.iorm.sf/slotsfile
2013-06-10T11:53:12.160Z: <test, 0> Error -1 in opening & reading the slot file2013-06-10T11:53:12.160Z: Couldn’t get a slot2013-06-10T11:53:12.160Z: <test, 0> Error in opening stat file for device: test.Ignoring this device.

In fact, this issue was referenced in a blog post by Frank Denneman here. Storage IO Control will not gather stats for this datastore. On further investigation we see the following permissions for the Storage I/O Control system files:

# ls -la /vmfs/volumes/test/.iormstats.sf
-rwxr-xr-x    1 nfsnobod root         16511 Jan 24 13:27 /vmfs/volumes/test/.iormstats.sf
# ls -la /vmfs/volumes/test/.iorm.sf/
drwxr-xr-x    1 nfsnobod root            19 Jun 10 11:56 .
drwxr-xr-x    1 nfsnobod root           160 Feb 20 10:58 ..
-r–r-x—    1 nfsnobod root           256 Jul  2  2012 slotsfile

Obviously, from the logs, there is a problem with permissions. Any attempt to change the owner of the files from nfsnobody to root produced the following error:

# chown root /vmfs/volumes/test/.iormstats.sf
chown: /vmfs/volumes/test/.iormstats.sf: Operation not permitted

The permissions for the NFS datastore were double checked and it was noted that the no root squash option was not in place. For NFS to work on vSphere, there must be no squashing of root privileges. Enclosed in a screen-shot on how to do this on an EMC Isilon array, which is what was used in-house to reproduce the issue:

After making this change, the permissions on the SIOC files did not change. They were still owned by user nfsnobody. However the files on the datastore could now be changed:

# ls -la /vmfs/volumes/test/.iormstats.sf
-rwxr-xr-x    1 nfsnobod root       16511 Jan 24 13:27 /vmfs/volumes/test/.iormstats.sf

# ls -la /vmfs/volumes/test/.iorm.sf/
drwxr-xr-x    1 nfsnobod root        19 Jun 10 11:56 .
drwxr-xr-x    1 nfsnobod root      160 Feb 20 10:58 ..
-r–r-x—    1 nfsnobod root      256 Jul  2  2012 slotsfile# chown root /vmfs/volumes/test/.iormstats.sf

# ls -la /vmfs/volumes/test/.iormstats.sf
-rwxr-xr-x    1 root     root         16511 Jun 10 12:07 /vmfs/volumes/test/.iormstats.sf

# chown -R  root /vmfs/volumes/test/.iorm.sf/

# ls -la /vmfs/volumes/test/.iorm.sf/
drwxr-xr-x    1 root     root            19 Jun 10 12:08 .
drwxr-xr-x    1 nfsnobod root           160 Feb 20 10:58 ..
-r–r-x—    1 root     root           384 Jun 10 12:07 slotsfile

After making this change, SIOC started working, as observed via the log files:

2013-06-10T12:07:37.786Z: stat file /vmfs/volumes//test/.iormstats.sf already exists.
2013-06-10T12:07:37.800Z: <test, 0> Successfully set shares for 9145 on test to 50000

Root Cause: It would appear that the root cause of this issue is that the NFS datastore was presented to the ESXi hosts without having the no_root_squash enabled, and turning on SIOC. Even after realizing that the no_root_squash option was missing and fixing this at the NFS array/NFS server end, the SIOC files were already created with the incorrect permissions. Only by manually changing the SIOC file permissions did we get SIOC to work.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

About the Author

Cormac Hogan

Cormac Hogan is a Senior Staff Engineer in the Office of the CTO in the Storage and Availability Business Unit (SABU) at VMware. He has been with VMware since April 2005 and has previously held roles in VMware’s Technical Marketing and Technical Support organizations. He has written a number of storage related white papers and have given numerous presentations on storage best practices and vSphere storage features. He is also the co-author of the “Essential Virtual SAN” book published by VMware Press.