iSCSI

NFS

Fiber Channel

FCoE

Description

iSCSI presents block devices to an ESXi host. Rather than accessing blocks from a local disk, the I/O operations are carried out over a network using a block access protocol. In case of iSCSI, remote blocks are accessed by encapsulating SCSI commands & data into TCP/IP packets. Support for iSCSI was introduced in ESX 3.0 back in 2006.

NFS (Network File System) presents file devices over a network to an ESXi host for mounting. The NFS server/array makes its local filesystems available to ESXi hosts. The ESXi hosts access the meta-data and files on the NFS array/server using a RPC-based protocol

VMware currently implements NFS version 3 over TCP/IP. VMware introduced support NFS in ESX 3.0 in 2006.

Fiber Channel presents block devices like iSCSI. Again the I/O operations are carried out over a network using a block access protocol. In FC, remote blocks are accessed by encapsulating SCSI commands & data into fiber channel frames.

One tends to see FC deployed in the majority of mission critical environments.

FC has been the only one of these 4 protocols supported on ESX since the beginning.

Fiber Channel over Ethernet also presents block devices, with I/O operations carried out over a network using a block access protocol. In this protocol, the SCSI commands and data are encapsulated into Ethernet frames. FCoE has many of the same characteristics of FC, except that the transport is Ethernet.

VMware Introduced support for HW FCoE in vSphere 4.x & SW FCoE in vSphere 5.0 back in 2011

Implementation Options

1.        NIC with iSCSI capabilities using Software iSCSI initiator & accessed using a VMkernel (vmknic) port

Or:

2.        Dependant Hardware iSCSI initiator

Or:

3.        Independent Hardware iSCSI initiator

Standard NIC accessed using a VMkernel port (vmknic)

Requires a dedicated Host Bus Adapter (HBA) (typically two for redundancy & multipathing)

1.        Hardware Converged Network Adapter (CNA)

Or:

2.        NIC with FCoE capabilities using Software FCoE initiator

Speed/Performance considerations

iSCSI can run over a 1Gb or a 10Gb TCP/IP network.

Multiple connections can be multiplexed into a single session, established between the initiator and target

VMware supports jumbo frames for iSCSI traffic, which can improve performance. Jumbo frames sends payloads larger than 1500. Support for jumbo frames with IP storage was introduced in ESX 4, but not on all initiators (KB 1007654 & KB  1009473). iSCSI can introduce overhead on a host’s CPU (encapsulating SCSI data into TCP/IP packets)

NFS can run over 1Gb or 10Gb over TCP/IP – NFS also supports UDP, but VMware's implementation does not & required TCP.

VMware supports jumbo frames for NFS traffic, which can improve performance in certain situations.

Support for jumbo frames with IP storage was introduced in ESX 4.

NFS can introduce overhead on a host’s CPU (encapsulating file I/O into TCP/IP packets)

Fiber Channel can run on 1Gb/2Gb/4Gb/8Gb & 16Gb, but 16Gb HBAs must be throttled to run at 8Gb in vSphere 5.0.

Buffer-to-Buffer credits & End-to-End credits throttle throughput to ensure lossless network

This protocol typically affects a host’s CPU the least as HBAs (required for FC) handles most of the processing (encapsulation of SCSI data into FC frames)

This protocol requires 10gb Ethernet.

The point to note with FCoE is that there is no IP encapsulation of the data like there is with NFS & iSCSI, which reduces some of the overhead/latency. FCoE is SCSI over Ethernet, not IP.

This protocol also requires jumbo frames since FC payloads are 2.2K in size and cannot be fragmented.

iSCSI

NFS

Fiber Channel

FCoE

Load Balancing

VMware’s Pluggable Storage Architecture (PSA) provides a Round-Robin Path Selection Policy which will distribute load across multiple paths to an iSCSI target. Better distribution of load with PSP_RR is achieved when multiple LUNs are accessed concurrently.

There is no load balancing per se on the current implementation of NFS as there is only a single session. Aggregate bandwidth can be configured by creating multiple paths to the NAS array, and accessing some datastores via one path, and other datastores via another.

VMware’s Pluggable Storage Architecture (PSA) provides a Round-Robin Path Selection Policy which will distribute load across multiple paths to an FC target. Better distribution of load with PSP_RR is achieved when multiple LUNs are accessed concurrently.

VMware’s Pluggable Storage Architecture (PSA) provides a Round-Robin Path Selection Policy which will distribute load across multiple paths to an FCoE target. Better distribution of load with PSP_RR is achieved when multiple LUNs are accessed concurrently.

Resilience

VMware’s PSA implements failover via its Storage Array Type Plugin (SATP) for all support iSCSI arrays. The preferred method to do this for SW iSCSI is with iSCSI Binding implemented, but it can be achieved with adding multiple targets on different subnets mapped to the iSCSI initiator.

NIC Teaming can be configured so that if one interface fails, another can take its place. However this is relying on a network failure and may not be able to handle error conditions occurring on the NFS array/server side.

VMware’s PSA implements failover via its Storage Array Type Plugin (SATP) for all support FC arrays

VMware’s PSA implements failover via its Storage Array Type Plugin (SATP) for all support FCoE arrays

Error checking

iSCSI uses TCP which resends dropped packets.

NFS uses TCP which resends dropped packets

Fiber Channel is implemented as a lossless network. This is achieved by throttling throughput at times of congestion using B2B and E2E credits

Fiber Channel over Ethernet requires a lossless network. This is achieved by the implementation of a Pause Frame mechanism at times of congestion.

Security

iSCSI implements the Challenge Handshake Authentication Protocol (CHAP) to ensure initiators and targets trust each other.

VLANs or private networks are highly recommended to isolate the iSCSI traffic from other traffic types.

VLANs or private networks are highly recommended to isolate the NFS traffic from other traffic types.

Some FC switches support the concepts of a VSAN to isolate parts of the storage infrastructure. VSANs are conceptually similar to VLANS.

Zoning between hosts and FC targets also offers a degree of isolation.

Some FCoE switches support the concepts of a VSAN to isolate parts of the storage infrastructure.

Zoning between hosts and FCoE targets also offers a degree of isolation.

iSCSI

NFS

Fiber Channel

FCoE

VAAI Primitives

Although VAAI primitives may be different from array to array, iSCSI devices can benefit from the full complement of block primitives:

·          Atomic Test/Set

·          Full Copy

·          Block Zero

·          Thin Provisioning

·          UNMAP

These primitives are built-in to ESXi, and require no additional software installed on the host.

Again, these vary for array to array. The VAAI primitives available on NFS devices are:

·          Full Copy (but not with Storage vMotion, only with cold migration)

·          Pre-allocate space (WRITE_ZEROs)

·          Clone offload using native snapshots

Note that for VAAI NAS, one requires a plug-in from the storage array vendor.

Although VAAI primitives may be different from array to array, FC devices can benefit from the full complement of block primitives:

·          Atomic Test/Set

·          Full Copy

·          Block Zero

·          Thin Provisioning

·          UNMAP

These primitives are built-in to ESXi, and require no additional software installed on the host.

Although VAAI primitives may be different from array to array, FCoE devices can benefit from the full complement of block primitives:

·          Atomic Test/Set

·          Full Copy

·          Block Zero

·          Thin Provisioning

·          UNMAP

These primitives are built-in to ESXi, and require no additional software installed on the host.

ESXi Boot from SAN

Yes

No

Yes

SW FCoE – No

HW FCoE (CNA) – Yes

RDM Support

Yes

No

Yes

Yes

Maximum Device Size

64TB

Refer to NAS array vendor or NAS server vendor for maximum supported datastore size.

Theoretical size is much larger than 64TB, but requires NAS vendor to support it.

64TB

64TB

Maximum number of devices

256

Default 8,

Maximum 256

256

256

Protocol direct to VM

Yes, via in-guest iSCSI initiator.

Yes, via in-guest NFS client.

No, but FC devices can be mapped directly to the VM with NPIV. This still requires RDM mapping to the VM first, and hardware must support NPIV (SW, HBA)

No

Storage vMotion Support

Yes

Yes

Yes

Yes

Storage DRS Support

Yes

Yes

Yes

Yes

Storage I/O Control Support

Yes, since vSphere 4.1

Yes, since vSphere 5.0

Yes, since vSphere 4.1

Yes, since vSphere 4.1

Virtualized MSCS Support

No. VMware does not support MSCS nodes built on VMs residing on iSCSI storage. However the use of software iSCSI initiators within guest operating systems configured with MSCS, in any configuration

supported by Microsoft, is transparent to ESXi hosts and there is no need for explicit support statements from

VMware. 

No. VMware does not support MSCS nodes built on VMs residing on NFS storage.

Yes, VMware supports MSCS nodes built on VMs residing on FC storage.

No. VMware does not support MSCS nodes built on VMs residing on FCoE storage.

iSCSI

NFS

Fiber Channel

FCoE

Ease of configuration

Medium – Setting up the iSCSI initiator requires some smarts, simply need the FDQN or IP address of the target. Some configuration for initiator maps and LUN presentation is needed on the array side. Once the target is discovered through a scan of the SAN, LUNs are available for datastores or RDMs.

Easy – Just need the IP or FQDN of the target, and the mount point. Datastore immediately appear once the host has been granted access from the NFS array/server side.

Difficult – Involves zoning at the FC switch level, and LUN masking at the array level once the zoning is complete. More complex to configure than IP Storage. Once the target is discovered through a scan of the SAN, LUNs are available for datastores or RDMs.

Difficult – Involves zoning at the FCoE switch level, and LUN masking at the array level once the zoning is complete. More complex to configure than IP Storage. Once the target is discovered through a scan of the SAN, LUNs are available for datastores or RDMs.

Advantages

No additional hardware necessary – can use already existing networking hardware components and iSCSI driver from VMware, so cheap to implement.

Well known and well understood protocol. Quite mature at this stage.

Admins with network skills should be able to implement.

Can be troubleshooted with generic network tools, such as wireshark.

No additional hardware necessary – can use already existing networking hardware components, so cheap to implement.

Well known and well understood protocol.

Also very mature.

Admins with network skills should be able to implement.

Can be troubleshooted with generic network tools, such as wireshark

Well known and well understood protocol.

Very mature, and trusted.

Found in majority of mission critical environments.

Enables converged networking, allowing the consolidation of network and storage traffic onto the same network via CNA – converged network adapter.

Using DCBx (Data Center Bridging protocol), FCoE has been made lossless even though it runs over Ethernet. DCBX does other things like enabling different traffic classes to run on the same network, but that is beyond the scope of this discussion.

Disadvantages

Inability to route with iSCSI Binding implemented.

Possible security issues, as there is no built in encryption, so care must be taken to isolate traffic (e.g. VLANs).

SW iSCSI can cause additional CPU overhead on the ESX host.

TCP can introduce latency for iSCSI.

Since there is only a single session per connection, configuring for maximum bandwidth across multiple paths needs some care and attention.

No PSA multipathing

Same security concerns as iSCSI since everything is transferred in clear text so care must be taken to isolate traffic (e.g. VLANs).

NFS is still version 3, which does not have the multipathing or security features of NFS v4 or NFS v4.1.

NFS can cause additional CPU overhead on the ESX host

TCP can introduce latency for NFS.

Still only runs at 8Gb which is slower than other networks (16Gb throttled to run at 8Gb in vSphere 5.0)

Needs dedicated HBA, FC switch, FC capable storage array which makes an FC implementation rather more expensive

Additional management overhead (e.g. switch zoning) is needed.

Could prove harder to troubleshoot compared to other protocols.

Rather new, and not quite as mature as other protocols at this time.

Requires a 10Gb lossless network infrastructure which can be expensive.

Cannot route between initiator and targets using native IP routing – instead it has to use protocols such as FIP (FCoE Initialization Protocol).

Could prove complex to troubleshoot/isolate issues with network and storage traffic using the same pipe.