This is an interesting one for those of you considering using Storage I/O Control (SIOC) on NFS datastores. We’ve come across an issue a couple of times whereby SIOC fails to enable on an NFS datastore due to a permissions issue. The following messages are seen in the SIOC logs on the ESXi hosts sharing the datastore:
2013-06-10T11:53:08.147Z: <test, 0> Error in opening stat file for device: test.Ignoring this device. 2013-06-10T11:53:12.159Z: stat file /vmfs/volumes//test/.iormstats.sf already exists. 2013-06-10T11:53:12.160Z: open /vmfs/volumes//test/.iorm.sf/slotsfile(0x10000042, 0x0) failed: Permission denied 2013-06-10T11:53:12.160Z: <test, 0> Giving UP Permission denied Error -1 opening SLOT file /vmfs/volumes//test/.iorm.sf/slotsfile 2013-06-10T11:53:12.160Z: <test, 0> Error -1 in opening & reading the slot file2013-06-10T11:53:12.160Z: Couldn’t get a slot2013-06-10T11:53:12.160Z: <test, 0> Error in opening stat file for device: test.Ignoring this device.
In fact, this issue was referenced in a blog post by Frank Denneman here. Storage IO Control will not gather stats for this datastore. On further investigation we see the following permissions for the Storage I/O Control system files:
# ls -la /vmfs/volumes/test/.iormstats.sf -rwxr-xr-x 1 nfsnobod root 16511 Jan 24 13:27 /vmfs/volumes/test/.iormstats.sf # ls -la /vmfs/volumes/test/.iorm.sf/ drwxr-xr-x 1 nfsnobod root 19 Jun 10 11:56 . drwxr-xr-x 1 nfsnobod root 160 Feb 20 10:58 .. -r–r-x— 1 nfsnobod root 256 Jul 2 2012 slotsfile
Obviously, from the logs, there is a problem with permissions. Any attempt to change the owner of the files from nfsnobody to root produced the following error:
# chown root /vmfs/volumes/test/.iormstats.sf chown: /vmfs/volumes/test/.iormstats.sf: Operation not permitted
The permissions for the NFS datastore were double checked and it was noted that the no root squash option was not in place. For NFS to work on vSphere, there must be no squashing of root privileges. Enclosed in a screen-shot on how to do this on an EMC Isilon array, which is what was used in-house to reproduce the issue:
After making this change, the permissions on the SIOC files did not change. They were still owned by user nfsnobody. However the files on the datastore could now be changed:
# ls -la /vmfs/volumes/test/.iormstats.sf -rwxr-xr-x 1 nfsnobod root 16511 Jan 24 13:27 /vmfs/volumes/test/.iormstats.sf
# ls -la /vmfs/volumes/test/.iormstats.sf -rwxr-xr-x 1 root root 16511 Jun 10 12:07 /vmfs/volumes/test/.iormstats.sf
# chown -R root /vmfs/volumes/test/.iorm.sf/
# ls -la /vmfs/volumes/test/.iorm.sf/ drwxr-xr-x 1 root root 19 Jun 10 12:08 . drwxr-xr-x 1 nfsnobod root 160 Feb 20 10:58 .. -r–r-x— 1 root root 384 Jun 10 12:07 slotsfile
After making this change, SIOC started working, as observed via the log files:
2013-06-10T12:07:37.786Z: stat file /vmfs/volumes//test/.iormstats.sf already exists. 2013-06-10T12:07:37.800Z: <test, 0> Successfully set shares for 9145 on test to 50000
Root Cause: It would appear that the root cause of this issue is that the NFS datastore was presented to the ESXi hosts without having the no_root_squash enabled, and turning on SIOC. Even after realizing that the no_root_squash option was missing and fixing this at the NFS array/NFS server end, the SIOC files were already created with the incorrect permissions. Only by manually changing the SIOC file permissions did we get SIOC to work.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter:@VMwareStorage
