As a follow-up to my previous post on the topic of Virtual SAN automation, VMware Virtual SAN Automated Deployments with PowerCLI, I want to continue the Virtual SAN automation theme and share another automated demonstration example.
This time, I want to highlight one of the primary use cases for Virtual SAN with the focus on availability and business continuity. Using PowerCLI, you can quickly perform the deployment and configuration of all of the necessary settings and artifacts required in a virtual infrastructure to configure a Virtual SAN cluster across multiple data centers. In this example, I have chosen one of the most complicated Virtua SAN deployment options from an implementation and maintenance perspective which is a Stretched Cluster between multiple data centers over layer 3 networks.
This type of Virtual SAN deployment is highly dependent on the physical network infrastructure and on the adequate configuration required to allow Virtual SAN communication over Layer 2 networks (Multicast) as well as routed networks (PIM). The preparation and configuration of the physical network infrastructure is typically the responsibility of a dedicated team of network engineers and
This type of Virtual SAN deployment is highly dependent on the physical network infrastructure and on the adequate configuration required to allow Virtual SAN communication over Layer 2 networks (Multicast) as well as routed networks (PIM). The preparation and configuration of the physical network infrastructure is typically the responsibility of a dedicated team of network engineers and therefore it is not something I’m trying to address here with PowerCLI. The assumption is that the physical network infrastructure has been configured and prepared to satisfy the networking requirements for Virtual SAN communication.
The sample code in the PowerCLI script featured here is ready for a vSphere Administrator to use as part of the deployment of a Virtual SAN Stretched Cluster over layer 3 networks. The code in the scripts performs the configuration and setup of all of the necessary vSphere related artifacts to form successfully an 8 node all-flash cluster across multiple sites. There are a couple of Virtual SAN 6.2 configuration functions that aren’t available in the latest release of PowerCLI 6.3 Release 1 and because of that, a couple of features will need to be enabled from the UI, but still with a couple of clicks and minimum complexity.
The great Alan Renouf promised me that missing PowerCLI cmdlets and functions are already in the works and will be added to PowerCLI soon. Beyond this little gap, this sample script can be used in any physical and vSphere infrastructure where the necessary physical network configuration settings for VMware Virtual SAN deployments across layer 3 have been satisfied and successfully implemented.
The following automated functions are performed with PowerCLI:
- Creation of the virtual datacenter
- Add the Virtual SAN Witness appliance to the data center and configure the Virtual SAN network IP address
- Creating a vSphere Cluster and add the hosts to the cluster
- Create the distributed switch and port groups for each host and network
- Add host related information such DNS, NTP, VMkernel Interfaces, VLANs
- Enable the Virtual SAN Network traffic services
- Add static routes on all hosts and Virtual SAN witness appliance necessary for intra-cluster
communication - Validate host communication and cross network connectivity by pinging the Virtual SAN network interface
- Claim the flash storage devices based on their model for their respective tiers were provided by Micron and form multiple disk groups per hosts
Once all of those functions are completed all that is left to do is going over the stretched cluster configuration workflow to define the locations, and add Virtual SAN witness to the cluster as the witness failure domain. All done view a few clicks. You will then be ready to begin the creation of virtual machine networks and deploying virtual machines. Before going into test mode or even production, don’t forget to complete the recommended HA and DRS configuration settings.
I used the latest version of PowerCLI which was released last week I recommend getting – PowerCLI 6.3 Release 1. I’ve included a demonstration of the script at work below to see it all working. I recommend watching the video in full screen more to clearly see the actions performed in the script just in case the font on the windows is too small to see.
You can copy the code below and use it as much as you like. I hope everyone finds this useful and for those PowerCLI pros out there feel free to provide comments and improvements. This script is great for all types of proof of concepts with minimal changes required. All you need to know and change is the IP addresses and settings to match the network settings relevant to the infrastructures where deploying Virtual SAN.
To get access to the PowerCLI code visit the link below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 |
#################################################################################### # # # VMware Virtual SAN -- All Flash Automated Deployment for Stretched Cluster # # # #################################################################################### #################################################################################### # # # The All-in-One and Ultimate Virtual SAN Streched Cluster Config. Script # # # # by Alan Renouf and Rawlinson # # # #################################################################################### # -- Add all VMware PowerCLI modules -- get-module -ListAvailable VMware* | Import-Module | Out-Null # -- Infrastructure Settings for scripts -- $VCNode = "vcenter-fqdn-or-ip" $VCPassword = "password" $ESXiUserName = "root" $ESXiPassword = 'password' $DCName = "octo-sabu" $CluName = "hci-stretched-cluster" $VDSName = "10G Switch" $PrimaryPortGroup = 3001 $SecondaryPortgroup = 3003 $WitnessPortgroup = 3002 $DNS = "10.142.7.21", "10.142.7.22" $NTP = "10.132.249.12", "10.132.249.28" $VMotionIP = "192.168.21." $VSANIP = "192.168." $cachingSSD = "S630DC-960" $CapacitySSD = "MICRON_M510DC_MT" $witness = "VSAN-witness-fqdn-or-ip" $witnessIP = "192.168.2.68" $WitnessSNM = "255.255.255.0" # -- Static Routes -- $PrimarySR = "192.168.1.0" $PrimaryGW = "192.168.1.253" $SecondarySR = "192.168.3.0" $SecondaryGW = "192.168.3.253" $WitnessSR = "192.168.2.0" $WitnessGW = "192.168.2.253" # -- Connect to vCenter -- Connect-viserver $VCNode -user $VCUserName -pass $VCPassword -WarningAction SilentlyContinue # -- Datacenter/Cluster Configuration -- # -- Create Datacenters -- Write-Host "Creating Datacenter: $DCName" -ForegroundColor Green $DC = New-Datacenter -Name $DCName -Location (Get-Folder Datacenters) # -- Adding and configuring the Virtual SAN Witness Appliance to vCenter Write-Host "Creating Witness Datacenter: witness-$($DCName)" -ForegroundColor Green $WDC = New-Datacenter -Name "Witness-$($DCName)" -Location (Get-Folder Datacenters) Write-Host "Adding Witness host $Witness" -ForegroundColor Green Add-VMHost -Name $witness -Location $WDC -User $ESXiUserName -Password $ESXiPassword -Force | Out-Null Write-Host "Adding Witness IP Address" -ForegroundColor Green $WVMK = get-vmhost $witness | Get-VMHostNetworkAdapter | Where { $_.dhcpEnabled -eq $true} $WVMK | Set-VMHostNetworkAdapter -IP $witnessIP -SubnetMask $WitnessSNM -Confirm:$false | Out-Null # -- Create Cluster -- Write-Host "Creating Cluster: $CluName" -ForegroundColor Green $CLU = New-Cluster -Name $CluName -Location ($DC) -DrsEnabled # -- Add Hosts to cluster -- 09..16 | Foreach { $num = $_ ; $newnum = "{0:D2}" -f $num Write-Host "Adding host hostname-$newnum.vmware.com" -ForegroundColor Green Add-VMHost -Name "hostname-$newnum.vmware.com" -Location $CluName -User $ESXiUserName -Password $ESXiPassword -Force | Out-Null } # -- Host Configuration -- $VMHosts = Get-VMHost | Sort Name # -- Add DNS/NTP and Enable iScsi Settings for the hosts -- Foreach ($vmhost in $vmhosts) { Write-Host "Configuring DNS and Domain Name on $vmhost" -ForegroundColor Green Get-VMHostNetwork -VMHost $vmhost | Set-VMHostNetwork -DNSAddress $DNS -Confirm:$false | Out-Null Write-Host "Configuring NTP Servers on $VMHost" -ForegroundColor Green Add-VMHostNTPServer -NtpServer $NTP -VMHost $VMHost -Confirm:$false -ErrorAction SilentlyContinue | FT | Out-Null Write-Host "Configuring NTP Client Policy on $VMHost" -ForegroundColor Green Get-VMHostService -VMHost $VMHost | where {$_.Key -eq "ntpd"} | Set-VMHostService -policy "on" -Confirm:$false | FT | Out-Null Write-Host "Restarting NTP Client on $VMHost" -ForegroundColor Green Get-VMHostService -VMHost $VMHost | where {$_.Key -eq "ntpd"} | Restart-VMHostService -Confirm:$false | FT | Out-Null } # -- Network Configuration -- # -- Create DVSwitch -- Write-Host "Creating VDSwitch: $VDSName" -ForegroundColor Green $VDS = New-VDSwitch -Name $VDSName -NumUplinkPorts 2 -Location $DC -Mtu 9000 -Version "6.0.0" # -- Create Portgroups -- Write-Host "Creating PortGroup: VSAN Network $PrimaryPortGroup" -ForegroundColor Green New-VDPortgroup -Name "VSAN Network $PrimaryPortGroup" -Vds $vds -VlanId $PrimaryPortGroup | Out-Null Write-Host "Creating PortGroup: VSAN Network $SecondaryPortGroup" -ForegroundColor Green New-VDPortgroup -Name "VSAN Network $SecondaryPortGroup" -Vds $vds -VlanId $SecondaryPortGroup | Out-Null Write-Host "Creating PortGroup: VSAN Network $WitnessPortGroup" -ForegroundColor Green New-VDPortgroup -Name "VSAN Network $WitnessPortGroup" -Vds $vds -VlanId $WitnessPortGroup | Out-Null Write-Host "Creating vMotion Network 3021" -ForegroundColor Green New-VDPortgroup -Name "vMotion Network 3021" -VDSwitch $vds -VlanId 3021 | Out-Null # -- Add Hosts to VDSWitch and Migrate pNIC to VDS (vmnic2/vmnic3) -- Foreach ($vmhost in ($DC | Get-VmHost)) { Write-Host "Adding $VMHost to $VDSName" -ForegroundColor Green $vds | Add-VDSwitchVMHost -VMHost $vmhost | Out-Null $vmhostNetworkAdapter = Get-VMHost $vmhost | Get-VMHostNetworkAdapter -Physical -Name vmnic2 Write-Host "Adding $VMHostNetworkAdapter to $VDSName" -ForegroundColor Green $vds | Add-VDSwitchPhysicalNetworkAdapter -VMHostNetworkAdapter $vmhostNetworkAdapter -Confirm:$false | Out-Null $vmhostNetworkAdapter = Get-VMHost $vmhost | Get-VMHostNetworkAdapter -Physical -Name vmnic3 Write-Host "Adding $VMHostNetworkAdapter to $VDSName" -ForegroundColor Green $vds | Add-VDSwitchPhysicalNetworkAdapter -VMHostNetworkAdapter $vmhostNetworkAdapter -Confirm:$false | Out-Null } # -- Set DVUplink2 to standby -- $TeamingPolicys = $vds | Get-VDPortgroup VSAN* | Get-VDUplinkTeamingPolicy Foreach ($Policy in $TeamingPolicys) { Write-Host "Setting Standby Uplink for $($Policy.VDPortGroup)" -ForegroundColor Green $Policy | Set-VDUplinkTeamingPolicy -StandbyUplinkPort "dvUplink2" | Out-Null } # -- Create vMotion VMKernel Ports for all hosts in DC -- foreach ($vmhost in ($DC | Get-VmHost)) { $HostIP = ($vmhost | Get-VMHostNetworkAdapter -Name vmk0).ip $LastO = $HostIP.Split(".")[3] $VSANNet = Get-VDPortGroup "VSAN Network $PrimaryPortGroup" $3rdO = ($VSANNet.Name).Substring(16) $CurrentvMotionIP = $vMotionIP + $LastO Write-Host "Adding vMotion Network Adapter to $VMHost with IP of $CurrentvMotionIP" -ForegroundColor Green New-vmhostnetworkadapter -VMHost $vmhost -PortGroup "vMotion Network 3021" -VirtualSwitch $vds -VMotionEnabled $true -IP $CurrentvMotionIP -SubnetMask "255.255.255.0" | Out-Null } #Adding Static Routes for Virtual SAN Streched Cluster # -- Primary site -- 09..12 | Foreach { $num = $_ $newnum = "{0:D2}" -f $num $VMHost = "hostname-$newnum.vmware.com" $HostIP = (Get-VMHost $vmhost | Get-VMHostNetworkAdapter -Name vmk0).ip $LastO = $HostIP.Split(".")[3] $VSANNet = Get-VDPortGroup "VSAN Network $PrimaryPortGroup" $3rdO = ($VSANNet.Name).Substring(16) $CurrentVSANIP = $VSANIP + $3rdO + "." + $LastO Write-Host "Adding $CurrentVSANIP to $($VSANNet.Name) and enabling VSAN traffic" -ForegroundColor Green $VSANVMK = New-vmhostnetworkadapter -VMHost $vmhost -PortGroup $VSANNet.Name -VirtualSwitch $vds -VsanTrafficEnabled $true -IP $CurrentVSANIP -SubnetMask "255.255.255.0" Write-Host "Adding Static Routes for primary Site to host hostname-$newnum.vmware.com" -ForegroundColor Green New-VMHostRoute -VMHost $VMHost -Destination $SecondarySR -Gateway $PrimaryGW -PrefixLength 24 -Confirm:$false | Out-Null New-VMHostRoute -VMHost $VMHost -Destination $WitnessSR -Gateway $PrimaryGW -PrefixLength 24 -Confirm:$false | Out-Null if (-not $1PrimaryIP) { $1PrimaryIP = (Get-VMHost $VMHost | Get-VMHostNetworkAdapter -Name $VMKnet).IP } } # -- Secondary Site -- 13..16 | Foreach { $num = $_ ; $newnum = "{0:D2}" -f $num $VMHost = "hostname-$newnum.vmware.com" $HostIP = (Get-VMHost $vmhost | Get-VMHostNetworkAdapter -Name vmk0).ip $LastO = $HostIP.Split(".")[3] $VSANNet = Get-VDPortGroup "VSAN Network $SecondaryPortGroup" $3rdO = ($VSANNet.Name).Substring(16) $CurrentVSANIP = $VSANIP + $3rdO + "." + $LastO Write-Host "Adding $CurrentVSANIP to $($VSANNet.Name) and enabling VSAN traffic" -ForegroundColor Green $VSANVMK = New-vmhostnetworkadapter -VMHost $vmhost -PortGroup $VSANNet.Name -VirtualSwitch $vds -VsanTrafficEnabled $true -IP $CurrentVSANIP -SubnetMask "255.255.255.0" Write-Host "Adding Static Routes for Secondary Site to host hostname-$newnum.vmware.com" -ForegroundColor Green New-VMHostRoute -VMHost $VMHost -Destination $PrimarySR -Gateway $SecondaryGW -PrefixLength 24 -Confirm:$false | Out-Null New-VMHostRoute -VMHost $VMHost -Destination $WitnessSR -Gateway $SecondaryGW -PrefixLength 24 -Confirm:$false | Out-Null if (-not $1SecondaryIP) { $1SecondaryIP = (Get-VMHost $VMHost | Get-VMHostNetworkAdapter -Name $VMKnet).IP } } # -- witness Site -- Write-Host "Adding Static Routes to Witness" -ForegroundColor Green New-VMHostRoute -VMHost $witness -Destination $PrimarySR -Gateway $WitnessGW -PrefixLength 24 -Confirm:$false | Out-Null New-VMHostRoute -VMHost $witness -Destination $SecondarySR -Gateway $WitnessGW -PrefixLength 24 -Confirm:$false | Out-Null # -- communication validation between all Hosts and networks -- # -- primary site -- 09..12 | Foreach { $num = $_ ; $newnum = "{0:D2}" -f $num $VMHost = "hostname-$newnum.vmware.com" $VMKnet = (Get-VMHostNetworkAdapter -VMHost $vmhost -PortGroup "VSAN*").Name Write-Host "Pinging $SecondarySR from $VMHost on $vmknet..." -ForegroundColor Green $esxcli = Get-ESXCLI -VMhost $vmhost $ping = $esxcli.network.diag.ping(2,$null,$null,$1SecondaryIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Secondary: $1SecondaryIP Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Secondary: $1SecondaryIP Failed" -ForegroundColor Red } Write-Host "Pinging $WitnessSR from $VMHost on $VMKnet..." -ForegroundColor Green $ping = $esxcli.network.diag.ping(2,$null,$null,$WitnessIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Witness: $WitnessSR Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Witness: $WitnessSR Failed" -ForegroundColor Red } } # -- secondary site -- 13..16 | Foreach { $num = $_ ; $newnum = "{0:D2}" -f $num $VMHost = "hostname-$newnum.vmware.com" $VMKnet = (Get-VMHostNetworkAdapter -VMHost $vmhost -PortGroup "VSAN*").Name Write-Host "Pinging $PrimarySR from $VMHost on $vmknet..." -ForegroundColor Green $esxcli = Get-ESXCLI -VMhost $vmhost $ping = $esxcli.network.diag.ping(2,$null,$null,$1PrimaryIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Primary: $1PrimaryIP Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Primary: $1PrimaryIP Failed" -ForegroundColor Red } Write-Host "Pinging $WitnessSR from $VMHost on $vmknet..." -ForegroundColor Green $ping = $esxcli.network.diag.ping(2,$null,$null,$WitnessIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Witness: $WitnessSR Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Witness: $WitnessSR Failed" -ForegroundColor Red } } # -- witness site -- $VMKnet = (Get-VMHostNetworkAdapter -VMHost $witness -PortGroup "VSAN*").Name Write-Host "Pinging $PrimarySR from $witness on $vmknet..." -ForegroundColor Green $esxcli = Get-ESXCLI -VMhost $witness $Ping = $esxcli.network.diag.ping(2,$null,$null,$1PrimaryIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Primary: $1PrimaryIP Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Primary: $1PrimaryIP Failed" -ForegroundColor Red } Write-Host "Pinging $SecondarySR from $witness on $vmknet..." -ForegroundColor Green $ping = $esxcli.network.diag.ping(2,$null,$null,$1SecondaryIP,$vmknet,$null,$null,$null,$null,$null,$null,$null,$null) | select -expand Summary if ($ping.Recieved -ge 1) { Write-Host "Ping of Secondary: $1SecondaryIP Tested OK" -ForegroundColor Yellow } Else { Write-Host "Ping of Secondary: $1SecondaryIP Failed" -ForegroundColor Red } # -- enable Virtual SAN Write-Host "Enabling VSAN in manual mode for Cluster: $CLU" -ForegroundColor Green $CLU = $CLU | Set-Cluster -VsanEnabled:$true -VsanDiskClaimMode Manual -Confirm:$false -ErrorAction SilentlyContinue # -- claim and tag caching and capacity flash devices Foreach ($vmh in ($DC | Get-VmHost)) { Write-Host "Finding disks for $($vmh)" $esxcli = Get-ESXCLI -VMhost $vmh 0..1 | Foreach { $DiskgroupNum = $_ +1 $Caching = ($esxcli.storage.core.device.list() | Where {$_.model -eq $cachingSSD})[$_] $Capacity = ($esxcli.storage.core.device.list() | Where {$_.model -eq $capacitySSD})[$_] Write-Host "Using $($caching.Vendor) - $($caching.Model) for Caching in Disk Group $Diskgroupnum" -foregroundColor Green Write-Host "Using $($Capacity.Vendor) - $($Capacity.Model) for Capacity in Disk Group $Diskgroupnum" -foregroundColor Green Write-Host "Tagging $($Capacity.Model) as Capacity" $capacitytag = $esxcli.vsan.storage.tag.add(($capacity.Device), "capacityFlash") Write-Host "Adding Storage devices to $($vmhost)" $adddisks = $esxcli.vsan.storage.add(($capacity.device), ($Caching.device)) if ($adddisks -eq "true") { Write-Host "Disks added" -ForegroundColor Green } Else { Write-Host "Error adding disks: $adddisks" -ForegroundColor Red } } Write-Host "Enable space efficiency sparse swap on $($vmh)" -ForegroundColor Green Get-AdvancedSetting -Entity $vmh -Name "VSAN.SwapThickProvisionDisabled" | Set-AdvancedSetting -Value 1 -Confirm:$false | out-null } # -- Virtual SAN cluster configuration validation If ($CLU.VSANEnabled){ Write-Host "VSAN cluster $($CLU.Name) created in $($CLU.VSANDiskClaimMode) configuration" -ForegroundColor Yellow Write-Host "The following Hosts and Disk Groups now exist:" Get-VsanDiskGroup | Select VMHost, Name | FT -AutoSize Write-Host "The following VSAN Datastore now exists:" Get-Datastore | Where {$_.Type -eq "vsan"} | Select Name, Type, FreeSpaceGB, CapacityGB } Else { Write-Host "Something went wrong, Virtual SAN not enabled" } |
I would like to thank the folks at Micron for being generous and providing the flash devices for the Virtual SAN infrastructure used for the creation of the demo. Also huge thanks to the one and only Mr. Alan Renouf for always wanting to help and collaborate.
-Enjoy
For future updates on Virtual SAN (VSAN), vSphere Virtual Volumes (vVols) and other Storage and Availability technologies, as well as vSphere Integrated OpenStack (VIO), and Cloud-Native Applications (CNA) be sure to follow me on Twitter: @PunchingClouds.
