Home > Blogs > VMware Consulting Blog > Tag Archives: vSAN

Tag Archives: vSAN

VMware Validated Design for SDDC 2.0 – Now Available

Jonathan McDonaldBy Jonathan McDonald

Recently I have been involved in a rather cool project inside VMware, aimed at validating and integrating all the different VMware products. The most interesting customer cases I see are related to this work because oftentimes products work independently without issue—but together can create unique problems.

To be honest, it is really difficult to solve some of the problems when integrating many products together. Whether we are talking about integrating a ticketing system, building a custom dashboard for vRealize Operations Manager, or even building a validation/integration plan for Virtual SAN to add to existing processes, there is always the question, “What would the experts recommend?”

The goal of this project is to provide a reference design for our products, called a VMware Validated Design. The design is a construct that:

  • Is built by expert architects who have many years of experience with the products as well as the integrations
  • Allow repeatable deployment of the end solution, which has been tested to scale
  • Integrates with the development cycle, so if there is an issue with the integration and scale testing, it can be identified quickly and fixed by the developers before the products are released.

All in all, this has been an amazing project that I’ve been excited to work on, and I am happy to be able to finally talk about it publicly!

Introducing the VMware Validated Design for SDDC 2.0

The first of these designs—under development for some time—is the VMware Validated Design for SDDC (Software-Defined Data Center). The first release was not available to the public and only internal to VMware, but on July 21, 2016, version 2.0 was released and is now available to everyone! This design builds not only the foundation for a solid SDDC infrastructure platform using VMware vSphere, Virtual SAN, and VMware NSX, but it builds on that foundation using the vRealize product suite (vRealize Operations Manager, vRealize Log Insight, vRealize Orchestrator, and vRealize Automation).

The VMware Validated Design for SDDC outcome requires a system that enables an IT organization to automate the provisioning of common, repeatable requests and to respond to business needs with more agility and predictability. Traditionally, this has been referred to as Infrastructure-as-a-Service (IaaS); however, the VMware Validated Design for SDDC extends the typical IAAS solution to include a broader and more complete IT solution.

The architecture is based on a number of layers and modules, which allows interchangeable components to be part of the end solution or outcome, such as the SDDC. If a particular component design does not fit the business or technical requirements for whatever reason, it should be able to be swapped out for another similar component. The VMware Validated Design for SDDC is one way of putting an architecture together that has been rigorously tested to ensure stability, scalability, and compatibility. Ultimately, however, the system is designed to ensure the desired outcome will be achieved.

The conceptual design is shown in the following diagram:

JMCDonald_VVD Conceptual Design

As you can see, the design brings a lot more than just implementation details. It includes many common “day two” operational tasks such as management and monitoring functions, business continuity, and security.

To simplify such a complex design, it has been broken up into:

  • A high-level Architecture Design
  • A Detailed Design with all the design decisions included
  • Implementation guidance.

Let’s take an in-depth look.

Continue reading

Virtualization and VMware Virtual SAN … the Old Married Couple

Don’t Mistake These Hyper-Converged Infrastructure Technologies as Mutually Exclusive

Jonathan McDonaldBy Jonathan McDonald

I have not posted many blogs recently as I’ve been in South Africa. I have however been hard at work on the latest release of VMware vSphere 6.0 Update 2 and VMware Virtual SAN 6.2. Some amazing features are included that will make life a lot easier and add some exciting new functionality to your hyper-converged infrastructure. I will not get into these features in this post, because I want to talk about one of the bigger non-technical questions that I get from customers and consultants alike. This is not one that is directly tied to the technology or architecture of the products. It is the idea that you can go into an environment and just do Virtual SAN, which from my experience is not true. I would love to know if your thoughts and experiences have shown you the same thing.

Let me first tell those of you who are unaware of Virtual SAN that I am not going to go into great depth about the technology. The key is that, as a platform, it is hyper-converged, meaning it is included with the ESXi hypervisor. This makes it radically simple to actually configure—and, more importantly, use—once it is up and running.

My hypothesis is that 80 to 90% of what you have to do to design for Virtual SAN focuses on the Virtualization design, and not so much on Virtual SAN.  This is not to say the Virtual SAN design is not important, but virtualization has to be integral to the design when you are building for it. To prove this, take a look at what the standard tasks are when creating the design for the environment:

  1. Hardware selection, racking, configuration of the physical hosts
  2. Selection and configuration of the physical network
  3. Software installation of the VMware ESXi hosts and VMware vCenter server
  4. Configuration of the ESXi hosts
    • Networking (For management traffic, and for VMware vSphere vMotion, at a minimum)
    • Disks
    • Features (VMware vSphere High Availability, VMware vSphere Distributed Resource Scheduler, VMware vSphere vMotion, at a minimum)
  5. Validation and testing of the configuration

If I add the Virtual SAN-specific tasks in, you have a holistic view of what is required in most greenfield configurations:

  1. Configuration of the Virtual SAN network
  2. Turning on Virtual SAN
  3. Creating new policies (optional, as the default is in place once configured)
  4. Testing Virtual SAN

As you can see, my first point shows that the majority of the work is actually virtualization and not Virtual SAN. In fact, as I write this, I am even more convinced of my hypothesis. The first three tasks alone are really the heavy hitters for time spent. As a consultant or architect, you need to focus on these tasks more than anything. Notice above where I mention “configure” in regards to Virtual SAN, and not installation; this is because it is already a hyper-converged element installed with ESXi. Once you get the environment up and running with ESXi hosts installed, Virtual SAN needs no further installation, simply configuration. You turn it on with a simple wizard, and, as long as you have focused on the supportability of the hardware and the underlying design, you will be up and running quickly. Virtual SAN is that easy.

Many of the arguments I get are interesting as well. Some of my favorites include:

  • “The customer has already selected hardware.”
  • “I don’t care about hardware.”
  • “Let’s just assume that the hardware is there.”
  • “They will be using existing hardware.”

My response is always that you should care a great deal about the hardware. In fact, this is by far the most important part of a Virtual SAN engagement. With Virtual SAN, if the hardware is not on the VMware compatibility list, then it is not supported. By not caring about hardware, you risk data loss and the loss of all VMware support.

If the hardware is already chosen, you should ensure that the hardware being proposed, added, or assumed as in place is proper. Get the bill of materials or the quote, and go over it line-by-line if that’s what’s needed to ensure that it is all supported.

Although the hardware selection is slightly stricter than with an average design, it is much the same as any traditional virtualization engagement in how you come to the situation. Virtual SAN Ready nodes are a great approach and make this much quicker and simpler, as they offer a variety of pre-configured hardware to meet the needs of Virtual SAN. Along with the Virtual SAN TCO Calculator it makes the painful process of hardware selection a lot easier.

Another argument I hear is “If I am just doing Virtual SAN, that is not enough time.” Yes, it is. It really, really is. I have been a part of multiple engagements for which the first five tasks above are already completely done. All we have to do is come in and turn on Virtual SAN. In Virtual SAN 6.2, this is made really easy with the new wizard:

JMcDonald_Configure VSAN

Even with the inevitable network issues (not lying here; every single time there is a problem with networking), environmental validation, performance testing, failure testing, testing virtual machine creation workflows, I have never seen it take more than a week to do this piece for a single cluster regardless of size of configuration. In many cases, after three days, everything is up and running and it is purely customer validation that is taking place. As a consultant or architect, don’t be afraid of the questions customers ask in regards to performance and failures. Virtual SAN provides mechanisms to easily test the environment as well as see as what “normal” is.

Here are two other arguments I hear frequently:

  • “We have never done this before.”
  • “We don’t have the skillset.”

These claims are probably not 100% accurate. If you have used VMware, or you are a VMware administrator, you are probably aware of the majority of what you have to do here. For Virtual SAN, specifically, this is where the knowledge needs to be grown. I suggest a training, or a review of VMworld presentations for Virtual SAN, to get familiar with this piece of technology and its related terminology. VMware offers training that will get you up to speed on hyper-converged infrastructure technologies, and the new features of VMware vSphere 6.0 Update Manager 2 and Virtual SAN 6.2.

For more information about free learnings, check out the courses below:

In addition, most of the best practices you will see are not unfamiliar since they are vCenter- or ESXi-related. Virtual SAN Health gives an amazing overview that is frequently refreshed, so any issues you may be seeing are reported here; this also takes a lot of the guess work out of the configuration tasks as you can see from the screenshot below, as many, if not all of, the common misconfigurations are shown.

JMcDonald_VSAN Health

In any case, I hope I have made the argument that Virtual SAN is mostly a virtualization design that just doesn’t use traditional SANs for storage.  Hyper-converged infrastructure is truly bringing change to many customers. This is, of course, just my opinion, and I will let you judge for yourself.

Virtual SAN has quickly become one of my favorite new technologies that I have worked with in my time at VMware, and I am definitely passionate about people using it to change the way they do business. I hope this helps in any engagements that you are planning as well as to prioritize and give a new perspective to how infrastructure is being designed.


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments

Virtual SAN Stretch Clusters – Real World Design Practices (Part 2)

Jonathan McDonaldBy Jonathan McDonald

This is the second part of a two blog series as there was just too much detail for a single blog. For Part 1 see: http://blogs.vmware.com/consulting/2016/01/virtual-san-stretch-clusters-real-world-design-practices-part-1.html .

As I mentioned at the beginning of the last blog, I want to start off by saying that all of the details here are based on my own personal experiences. It is not meant to be a comprehensive guide to setting up stretch clustering for Virtual SAN, but rather a set of pointers to show the type of detail most commonly asked for. Hopefully it will help you prepare for any projects of this type.

Continuing on with the configuration, the next set of questions regarded networking!

Networking, Networking, Networking

With sizing and configuration behind us, the next step was to enable Virtual SAN and set up the stretch clustering. As soon as we turned it on, however, we got the infamous “Misconfiguration Detected” message for the networking.

In almost all engagements I have been a part of, this has been a problem, even though the networking team said it was already set up and configured. This always becomes a fight, but it gets easier with the new Health UI Interface and multicast checks. Generally, when multicast is not configured properly, you will see something similar to the screenshot shown below.

JMcDonald VSAN Pt2 (1)

It definitely makes the process of going to the networking team easier. The added bonus is there are no messy command line syntaxes needed to validate the configuration. I can honestly say the health interface for Virtual SAN is one of the best features introduced for Virtual SAN!

Once we had this configured properly the cluster came online and we were able to configure the cluster, including stretch clustering, the proper vSphere high availability settings and the affinity rules.

The final question that came up on the networking side was about the recommendation that L3 is the preferred communication mechanism to the witness host. The big issue when using L2 is the potential that traffic could be redirected through the witness in the case of a failure, which has a substantially lower bandwidth requirement. A great description of this concern is in the networking section of the Stretched Cluster Deployment Guide.

In any case, the networking configuration is definitely more complex in stretched clustering because the networking across multiple sites. Therefore, it is imperative that it is configured correctly, not only to ensure that performance is at peak levels, but to ensure there is no unexpected behavior in the event of a failure.

High Availability and Provisioning

All of this talk finally led to the conversation about availability. The beautiful thing about Virtual SAN is that with the “failures to tolerate” setting, you can ensure there are between one and three copies of the data available, depending on what is configured in the policy. Gone are the long conversations of trying to design this into a solution with proprietary hardware or software.

A difference with stretch clustering is that the maximum “failures to tolerate” is one. This is because we have three fault domains: the two sites and the witness. Logically, when you look at it, it makes sense: more than that is not possible with only three fault domains. The idea here is that there is a full copy of the virtual machine data at each site. This allows for failover in case an entire site fails as components are stored according to site boundaries.

Of course, high availability (HA) needs to be aware of this. The way this is configured from a vSphere HA perspective is to assign the percentage of cluster resources allocation policy and set both CPU and memory to 50 percent:
JMcDonald VSAN Pt2 (2)

This may seem like a LOT of resources, but when you think of it from a site perspective, it makes sense; if you have an entire site fail, resources in the failed site will be able to restart without issues.

The question came up as to whether or not we allow more than 50 percent to be assigned. Yes, we can set it to use more than half consumed, but there might be an issue if there is a failure, as all virtual machines may not start back up. This is why it is recommended that 50 percent of resources be reserved. If you do want to configure a utilization of more than 50 percent of the resources for virtual machines, it is still possible, but not recommended. This configuration generally consists of setting a priority on the most important virtual machines so HA will start up as many as possible, starting with the most critical ones. Personally, I recommend not setting above 50 percent for a stretch cluster.

An additional question came up about using host and virtual machine affinity rules to control the placement of virtual machines. Unfortunately, the assignment to these groups is not easy during provisioning process and did not fit easily into the virtual machine provisioning practices that were used in the environment. vSphere Distributed Resource Scheduler (DRS) does a good job ensuring balance, but more control was needed rather than just relying on DRS to balance the load. The end goal was that during provisioning, placement in the appropriate site could be done automatically by staff.

This discussion boiled down to the need for a change to provisioning practices. Currently, it is a manual configuration change, but it is possible to use automation such as vRealize Orchestrator to automate deployment appropriately. This is something to keep in mind when working with customers to design a stretch cluster, as changes to provisioning practices may be needed.

Failure Testing

Finally, after days of configuration and design decisions, we were ready to test failures. This is always interesting because the conversation always varies between customers. Some require very strict testing and want to test every scenario possible, while others are OK doing less. After talking it over we decided on the following plan:

  • Host failure in the secondary site
  • Host failure in the primary site
  • Witness failure (both network and host)
  • Full site failure
  • Network failures
    • Witness to site
    • Site to site
  • Disk failure simulation
  • Maintenance mode testing

This was a good balance of tests to show exactly what the different failures look like. Prior to starting, I always go over the health status windows for Virtual SAN as it updates very quickly to show exactly what is happening in the cluster.

The customer was really excited about how seamlessly Virtual SAN handles errors. The key is to operationally prepare and ensure the comfort level is high with handling the worst-case scenario. When starting off, host and network failures are always very similar in appearance, but showing this is important; so I suggested running through several similar tests just to ensure that tests are accurate.

As an example, one of the most common failure tests requested (which many organizations don’t test properly) is simulating what happens if a disk fails in a disk group. Simply pulling a disk out of the server does not replicate what would happen if a disk actually fails, as a completely different mechanism is used to detect this. You can use the following commands to properly simulate a disk actually failing by injecting an error.  Follow these steps:

  1. Identify the disk device in which you want to inject the error. You can do this by using a combination of the Virtual SAN Health User Interface, and running the following command from an ESXi host and noting down the naa.<ID> (where <ID> is a string of characters) for the disk:
     esxcli vsan storage list
  2. Navigate to /usr/lib/vmware/vsan/bin/ on the ESXi host.
  3. Inject a permanent device error to the chosen device by running:
    python vsanDiskFaultInjection.pyc -p -d <naa.id>
  4. Check the Virtual SAN Health User Interface. The disk will show as failed, and the components will be relocated to other locations.
  5. Once the re-sync operations are complete, remove the permanent device error by running:
    python vsanDiskFaultInjection.pyc -c -d <naa.id>
  6. Once completed, remove the disk from the disk group and uncheck the option to migrate data. (This is not a strict requirement because data has already been migrated as the disk officially failed.)
  7. Add the disk back to the disk group.
  8. Once this is complete, all warnings should be gone from the health status of Virtual SAN.
    Note: Be sure to acknowledge and reset any alarms to green.

After performing all the tests in the above list, the customer had a very good feeling about the Virtual SAN implementation and their ability to operationally handle a failure should one occur.

Performance Testing

Last, but not least, was performance testing. Unfortunately, while I was onsite for this one, the 10G networking was not available. I would not recommend using a gigabit network for most configurations, but since we were not yet in full production mode, we did go through many of the performance tests to get a baseline. We got an excellent baseline of what the performance would look like with the gigabit network.

Briefly, because I could write an entire book on performance testing, the quickest and easiest way to test performance is with the Proactive Tests menu which is included in Virtual SAN 6.1:

JMcDonald VSAN Pt2 (3)

It provides a really good mechanism to test different types of workloads that are most common – all the way from a basic test, to a stress test. In addition, using IOmeter for testing (based on environmental characteristics) can be very useful.

In this case, to give you an idea of performance test results, we were pretty consistently getting a peak of around 30,000 IOPS with the gigabit network with 10 hosts in the cluster. Subsequently, I have been told that once the 10G network was in place, this actually jumped up to a peak of 160,000 IOPS for the same 10 hosts. Pretty amazing to be honest.

I will not get into the ins and outs of testing, as it very much depends on the area you are testing. I did want to show, however, that it is much easier to perform a lot of the testing this way than it was using the previous command line method.

One final note I want to add in the performance testing area is that one of the key things (other than pure “my VM goes THISSSS fast” type tests), is to test the performance of rebalancing in the case of maintenance mode, or failure scenarios. This can be done from the Resyncing Components Menu:

JMcDonald VSAN Pt2 (4)

Boring by default perhaps, but when you either migrate data in maintenance mode, or change a storage policy, you can see what the impact will be to resync components. It will either show when creating an additional disk stripe for a disk, or when fully migrating data off the host when going into maintenance mode. The compliance screen will look like this:

JMcDonald VSAN Pt2 (5)

This represents a significant amount of time, and is incredibly useful when testing normal workloads such as when data is migrated during the enter maintenance mode workflow. Full migrations of data can be incredibly expensive, especially if the disks are large, or if you are using gigabit rather than 10G networks. Oftentimes, convergence can take a significant amount of time and bandwidth, so this allows customers to plan for the amount of data to be moved while in or maintenance mode, or in the case of a failure.

Well, that is what I have for this blog post. Again, this is obviously not a conclusive list of all decision points or anything like that; it’s just where we had the most discussions that I wanted to share. I hope this gives you an idea of the challenges we faced, and can help you prepare for the decisions you may face when implementing stretch clustering for Virtual SAN. This is truly a pretty cool feature and will provide an excellent addition to the ways business continuity and disaster recovery plans can be designed for an environment.


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments

Virtual SAN Stretch Clusters – Real World Design Practices (Part 1)

Jonathan McDonaldBy Jonathan McDonald

This is part one of a two blog series as there was just too much detail for a single blog. I want to start off by saying that all of the details here are based on my own personal experiences. It is not meant to be a comprehensive guide for setting up stretch clustering for Virtual SAN, but a set of pointers to show the type of detail that is most commonly asked for. Hopefully it will help prepare you for any projects that you are working on.

Most recently in my day-to-day work I was asked to travel to a customer site to help with a Virtual SAN implementation. It was not until I got on site that I was told that the idea for the design was to use the new stretch clustering functionality that VMware added to the Virtual SAN 6.1 release. This functionality has been discussed by other folks in their blogs, so I will not reiterate much of the detail from them here. In addition, the implementation is very thoroughly documented by the amazing Cormac Hogan in the Stretched Cluster Deployment Guide.

What this blog is meant to be is a guide to some of the most important design decisions that need to be made. I will focus on the most recent project I was part of; however, the design decisions are pretty universal. I hope that the detail will help people avoid issues such as the ones we ran into while implementing the solution.

A Bit of Background

For anyone not aware of stretch clustering functionality, I wanted to provide a brief overview. Most of the details you already know about Virtual SAN still remain true. What it really amounts to is a configuration that allows two sites of hosts connected with a low latency link to participate in a virtual SAN cluster, together with an ESXi host or witness appliance that exists at a third site. This cluster is an active/active configuration that provides a new level of redundancy, such that if one of the two sites has a failure, the other site will immediately be able to recover virtual machines at the failed site using VMware High Availability.

The configuration looks like this:

JMcDonald Stretched Virtual SAN Cluster 1

This is accomplished by using fault domains and is configured directly from the fault domain configuration page for the cluster:

JMcDonald Stretched Virtual SAN Cluster 2

Each site is its own fault domain which is why the witness is required. The witness functions as the third fault domain and is used to host the witness components for the virtual machines in both sites. In Virtual SAN Stretched Clusters, there is only one witness host in any configuration.

JMcDonald Stretched Virtual SAN Cluster 3

For deployments that manage multiple stretched clusters, each cluster must have its own unique witness host.

The nomenclature used to describe a Virtual SAN Stretched Cluster configuration is X+Y+Z, where X is the number of ESXi hosts at data site A, Y is the number of ESXi hosts at data site B, and Z is the number of witness hosts at site C.

Finally, with stretch clustering, the current maximum configuration is 31 nodes (15 + 15 + 1 = 31 nodes). The minimum supported configuration is 1 + 1 + 1 = 3 nodes. This can be configured as a two-host virtual SAN cluster, with the witness appliance as the third node.

With all these considerations, let’s take a look at a few of the design decisions and issues we ran into.

Hosts, Sites and Disk Group Sizing

The first question that came up—as it almost always does—is about sizing. This customer initially used the Virtual SAN TCO Calculator for sizing and the hardware was already delivered. Sounds simple, right? Well perhaps, but it does get more complex when talking about a stretch cluster. The questions that came up regarded the number of hosts per site, as well as how the disk groups should be configured.

Starting off with the hosts, one of the big things discussed was the possibility of having more hosts in the primary site than in the secondary. For stretch clusters, an identical number of hosts in each site is a requirement. This makes it a lot easier from a decision standpoint, and when you look closer the reason becomes obvious: with a stretched cluster, you have the ability to fail over an entire site. Therefore, it is logical to have identical host footprints.

With disk groups, however, the decision point is a little more complex. Normally, my recommendation here is to keep everything uniform. Thus, if you have 2 solid state disks and 10 magnetic disks, you would configure 2 disk groups with 5 disks each. This prevents unbalanced utilization of any one component type, regardless of whether it is a disk, disk group, host, network port, etc. To be honest, it also greatly simplifies much of the design, as each host/disk group can expect an equal amount of love from vSphere DRS.

In this configuration, though, it was not so clear because one additional disk was available, so the division of disks cannot be equal. After some debate, we decided to keep one disk as a “hot spare,” so there was an equal number of disk groups—and disks per disk group—on all hosts. This turned out to be a good thing; see the next section for details.

In the end, much of this is the standard approach to Virtual SAN configuration, so other than site sizing, there was nothing really unexpected.

Booting ESXi from SD or USB

I don’t want to get too in-depth on this, but briefly, when you boot an ESXi 6.0 host from a USB device or SD card, Virtual SAN trace logs are written to RAMdisk, and the logs are not persistent. This actually serves to preserve the life of the device as the amount of data being written can be substantial. When running in this configuration these logs are automatically offloaded to persistent media during shutdown or system crash (PANIC). If you have more than 512 GB of RAM in the hosts, you are unlikely to have enough space to store this volume of data because these devices are not generally this large. Therefore, logs, Virtual SAN trace logs, or core dumps may be lost or corrupted because of insufficient space, and the ability to troubleshoot failures will be greatly limited.

So, in these cases it is recommended to configure a drive for the core dump and scratch partitions. This is also the only supported method for handling Virtual SAN traces when booting an ESXi from a USB stick or SD card.

That being said, when we were in the process of configuring the hosts in this environment, we saw the “No datastores have been configured” warning message pop up – meaning persistent storage had not been configured. This triggered the whole discussion; the error is similar to the one in the vSphere Web Client.
In the vSphere Client, this error also comes up when you click to the Configuration tab:

JMcDonald Stretched Virtual SAN Cluster 4

In the vSphere Client, this error also comes up when you click to the Configuration tab:

JMcDonald Stretched Virtual SAN Cluster 5

The spare disk turned out to be useful because we were able to use it to configure the ESXi scratch dump and core dump partitions. This is not to say we were seeing crashes, or even expected to; in fact, we saw no unexpected behavior in the environment up to this point. Rather, since this was a new environment, we wanted to ensure we’d have the ability to quickly diagnose any issue, and having this configured up-front saves significant time in support. This is of course speaking from first-hand experience.

In addition, syslog was set up to export logs to an external source at this time. Whether using the syslog service that is included with vSphere, or vRealize Log Insight (amazing tool if you have not used it), we were sure to have the environment set up to quickly identify the source of any problem that might arise.

For more details on this, see the following KB articles for instructions:

I guess the lesson here is that when you are designing your virtual SAN cluster, make sure you remember that having persistence available for logs, traces and core dumps is a best practice. If you have a large memory configuration, this is the easiest way to install ESXi and the scratch/core dump partitions to a hard drive. This also simplifies post-installation tasks, and will ensure you can collect all the information support might require to diagnose issues.

 Witness Host Placement

The witness host was the next piece we designed. Officially, the witness must be in a distinct third site in order to properly detect failures. It can either be a full host or a virtual appliance residing outside of the virtual SAN cluster. The cool thing is that if you use an appliance, it actually appears differently in the Web client:

JMcDonald Stretched Virtual SAN Cluster 6

For the witness host in this case, we decided to use the witness appliance rather than a full host. This way, it could be migrated easily because the networking was not set up to the third site yet. As a result, for the initial implementation while I was onsite, the witness was local to one of the sites, and would be migrated as soon as the networking was set up. This is definitely not a recommended configuration, but for testing—or for a non-production proof-of-concept—it does work. Keep in mind, that a site failure may not be properly detected unless the cluster is properly configured.

With this, I conclude Part 1 of this blog series; hopefully, you have found this useful. Stay tuned for Part 2!


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments

 

VMworld 2015 US Sneak Peek: Successful Virtual SAN Evaluation/Proof-Of-Concepts

Julienne_PhamBy Julienne Pham

This is the first in a series of previews of our VMware Professional Services speaking sessions at VMworld 2015 US starting August 30th in San Francisco. Our Technical Architects will present “deep-dive” sessions in their areas of expertise. Read these previews, register early and enjoy.

Sneak Peek of Session STO4572: Successful Virtual SAN Evaluation/Proof-Of-Concepts

This is an update to last year’s Virtual SAN proof-of-concept (POC) talk. A lot has changed in the last year, and the idea of this session is to fill you in on all the potential “gotchas” you might encounter when trying to evaluate VMware Virtual SAN.

Cormac Hogan, Corporate Storage Architect, and Julienne Pham, Technical Solution Architect of VMware, will cover everything you need to know, including how to conduct various failure scenarios, and get the best performance. Thinking about deploying Virtual SAN? Then this session is for you.

This session will share key tips on how to conduct a successful Virtual SAN proof-of-concept. It will show you how to correctly and properly set up a Virtual SAN environment (hosts, storage and networking), verify it is operating correctly, and then show you how to test the Virtual SAN full functionality. This session will also highlight how to verify that VM Storage Policies are working correctly, as they are an integral part of SDS and Virtual SAN.

We will also discuss how a Virtual SAN handles failures, and how to test if it is handling events correctly. In addition, it will cover numerous monitoring tools that can be used during a POC, such as the Ruby vSphere Console, VSAN, Observer Web-based analysis tools, and the new Virtual SAN Health Service plug-in. After attending this session, you will be empowered to implement your own Virtual SAN POC.

 Learn more at VMworld 2015 US in San Francisco

STO4572: Successful Virtual SAN Evaluation/Proof-Of-Concepts

Monday, Aug. 31, 8:30 – 9:30 AM

 Speakers:

Cormac Hogan, Corporate Storage Architect, VMware

Julienne Pham, Technical Solution Architect, VMware


Julienne Pham is a Technical Solution Architect for the Professional Services Engineering team. She is specialised on SRM and core storage. Her focus is on VIO and BCDR space.

Copying App Volumes AppStacks Between Environments That Use vSAN Storage

JeffSmallBy Jeffrey Davidson

There is an issue copying App Volumes AppStacks (VMDK files) if using Secure Copy (SCP) from one environment to another when using VSAN storage. This is not an App Volumes problem; it is related to the way VSAN stores VMDK data.

Clients will often want to create AppStacks in a test environment, then copy those AppStacks to a production environment, and finally import them into App Volumes. In situations where any of those environments use VSAN storage, you will not be able to copy (SCP) AppStacks (specifically VMDK files) between environments.

In this blog entry I will discuss a workaround to this issue, using an example in which the client has two VSAN environments (DEV and PROD), and needs to copy VMDK files between them.

The VMDK files created by App Volumes are nothing special and are traditionally comprised of two files.

What we normally identify as <filename>.vmdk is a type of header/metadata file. Meaning it only holds information regarding the geometry of the virtual disk and, as such, references a file that contains the actual data.

The file referenced is often called a “flat” file; this file contains the actual data of the VMDK. We can identify this file as it has the naming pattern of <filename>-flat.vmdk

On traditional block level storage these two files are normally stored together in the same folder, as shown in the example screenshot below.

JDavidson1

But VSAN storage is different; if you look at the contents of the “header” file you see something similar to the screenshot below. In this screenshot, the section in red is normally a reference to a “flat” file (example: Adobe_Acrobat_Reader -flat.vmdk). In the case where VSAN is the storage platform, we see something different. The screenshot below shows a reference to a VSAN device.

JDavidson2

VSAN storage employs object-level storage, which is different from traditional block-level storage. The VSAN objects are managed through a storage policy which, for example, can allow for greater redundancy for some virtual machines over others. Because the reference in the VMDK file points to a VSAN DOM object, it cannot be copied through traditional means (SCP).

To work around this issue you will need traditional block-level storage which acts as a “middle man” to allow the SCP copy of VMDK files between environments. You will also need SSH access enabled on one host in each environment.

The first step is to clone the VMDK you wish to copy from the VSAN volume to the traditional storage volume. Once on traditional storage you will be able to copy (SCP) the two VMDK files directly to a host in a different VSAN environment. After you have copied (SCP) the VMDK files to a destination VSAN environment, you will need to perform a clone action to re-integrate the VMDK with VSAN as a storage object, so it can be protected properly with VSAN.

The diagram below is an overview of the process to copy AppStack (VMDK) files between VSAN environments.

JDavidson3

The example below shows the commands required to copy an App Volumes AppStack (VMDK) between environments that use VSAN storage. Before executing these commands you should create a staging area in each environment where AppStacks will be copied temporarily before being copied between hosts for getting re-integrated in the destinations’ VSAN storage.

For example:

In the source environment, create the folder <path to block level storage>/AppVolumes_Staging

In the destination environment, create the folder <path to cloud volumes root folder>/cloudvolumes/staging

Step 1:

SSH into the host where the AppStack currently resides.

Execute the following command to clone the AppStack to block-level storage. Note that after you execute this command there are two files on the block-level storage. One is the header file, and the other is the “flat” file, which was previously integrated with VSAN as a storage object.

vmkfstools -d thin -i <VSAN path to App Stack>/cloudvolumes/apps/<filename>.vmdk <path to block level storage>/AppVolumes_Staging/<filename>.vmdk

Example:

vmkfstools -d thin -i /vmfs/volumes/vsan:4a65d9cbe47d44af-80f530e9e2b98ac5/76f05055-98b3-07ab-ef94-002590fd9036/apps/<filename>.vmdk /vmfs/volumes/54e5e55d-97561a60-50de-002590fd9036/AppVolumes_Staging/<filename>.vmdk


Step 2:

Execute the following commands to copy (SCP) an AppStack from one environment to another.

scp <path to vmdk clone on block level storage>/<filename>.vmdk root@<esxi mgt IP>:<path to staging folder>/<filename>.vmdk

scp <path to vmdk “flat” file clone on block level storage>/<filename>-flat.vmdk root@<esxi mgt IP>:<path to staging folder>/<filename>-flat.vmdk

Example:

scp /vmfs/volumes/54e5e55d-97561a60-50de-002590fd9036/AppVolumes_Staging/<filename>.vmdk root@10.10.10.10:/vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/Staging/<filename>.vmdk

scp /vmfs/volumes/54e5e55d-97561a60-50de-002590fd9036/AppVolumes_Staging/<filename>-flat.vmdk root@10.10.10.10:/vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/Staging/<filename>-flat.vmdk


Step 3:

Run the commands below to delete the AppStack from the staging folder on the source environment.

rm <path to staging folder>/<filename>.vmdk

rm <path to staging folder>/<filename>-flat.vmdk

Example:

rm /vmfs/volumes/54e5e55d-97561a60-50de-002590fd9036/AppVolumes_Staging/<filename>.vmdk

rm /vmfs/volumes/54e5e55d-97561a60-50de-002590fd9036/AppVolumes_Staging/<filename>-flat.vmdk


Step 4:

SSH into the host where the AppStack has been copied to. In this example the host IP address is 10.10.10.10.

Run the command below to clone the copied AppStack from the staging folder to the App Volumes “apps” folder, and re-integrate the VMDK into VSAN as a storage object.

vmkfstools -d thin -i <path to staging folder>/<filename>.vmdk <path to cloud volumes root folder>/cloudvolumes/apps/<filename>.vmdk

Example:

vmkfstools -d thin -i /vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/Staging/<filename>.vmdk /vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/apps/<filename>.vmdk


Step 5:

Run the commands below to delete the AppStack from the staging folder on the destination environment.

rm <path to staging folder>/<filename>.vmdk

rm <path to staging folder>/<filename>-flat.vmdk

Example:

rm /vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/Staging/<filename>.vmdk

rm /vmfs/volumes/vsan:265d91daeb2841db-82d3d8026326af8e/6efbac55-f2f7-f86a-033f-0cc47a59dc1c/Staging/<filename>-flat.vmdk
After completing these steps, you will have successfully copied a VMDK from one VSAN storage platform to another.

App Volumes also creates a “metadata” file during the creation of an AppStack, as shown in the screenshot below.

JDavidson4

The “metadata” file is a “text” file and should be copied to the destination environment so the AppStack (VMDK) can be imported into the destination App Volumes instance. Because this is a “text” file, it can be copied (SCP) without the cloning process and need for block-level storage as described in steps 1–5 above.


Jeffrey Davidson, Senior Consultant, VMware EUC. Jeffrey has over 15 years of IT experience and joined VMware in 2014. He is also a VCP5-DCV and VCP5-DT. He is a strong advocate of virtualization technologies, focusing on operational readiness with customers.

Top 5 Tips When Considering a VMware Virtual SAN Storage Solution

By Mark Moretti

Is a software-defined storage platform right for you? How do you approach evaluating a virtualized storage environment? What are the key considerations to keep in mind? What are VMware customers doing? And, what are the experts recommending?

We recently asked our VMware Professional Services consultants these questions. We asked them to provide us with some of the key tips they provide to their best customers. What did we get? A short list of “tips” that identify how to approach the consideration process for a VMware Virtual SAN solution. It’s not a trivial decision. Your IT decisions are not made in a vacuum. You have existing compute and storage infrastructure, so you need to know what the impact of your decisions will be—in advance.

Read this short list of recommendations, share it with your staff and engage in a conversation on transforming your storage infrastructure.

Top Tips 1

Read now: Top 5 Tips When Considering VMware’s Virtual SAN Storage Solution


Mark Moretti is a Senior Services Marketing Manager for VMware.

Understanding View Disposable Disks

Travis WoodBy Travis Wood, VCDX-97

When VMware introduced Linked-Clones in View 4.5 there was a new type of disk included called the Disposable Disk. The purpose of this disk was to redirect certain volatile files away from the OS Disk to help reduce linked-clone growth.  I have read a lot of designs that utilize disposable disks but it has become clear that there is a lot of confusion and misunderstanding about what they do and exactly how they function.  This confusion is highlighted in a View whitepaper called View Storage Considerations which describes disposable disks as:

Utilizing the disposable disk allows you to redirect transient paging and temporary file operations to a VMDK hosted on an alternate datastore. When the virtual machine is powered off, these disposable disks are deleted.

The three elements from this paragraph I want to demystify are:

  1. What is redirected to the disposable disk?
  2. Where are disposable disks hosted?
  3. When are disposable disks deleted/refreshed?

What is redirected?

By default there are three elements that are redirected to the disposable disk.  The first is the Windows swap file, View Composer will redirect the Swap file from C: to the disposable disk. It is recommended to set this to a specific size to make capacity planning easier.

 

TWood1

 

The other elements that are redirected are the System Environment Variables TMP and TEMP.  By default, the User TEMP and TMP Environment Variables are NOT redirected.  However it is highly recommended to remove the User TEMP and TMP Environment variables, if this is done then Windows will use the System Variables instead and the user temporary files will then be redirected to the disposable disk.

TWood4

 

 

Where is the disposable disk stored?

There is a common misconception that like the User Data Disk, the Disposable Disk can be redirected to a different tier.  This is not the case and the Disposable Disk is always stored with the OS Disk.  In later versions of View you can choose the drive letter within the GUI for the Disposable Disk to avoid conflicts with mapped drives, but this setting and the size are the only customizations you can make to the disposable disk.

When is the disposable disk refreshed?

This is the question that tends to cause the most confusion.  Many people I have spoken to have said that it is refreshed when the user logs off, whilst others say it’s on reboot.  The Disposable Disk is actually only refreshed when View powers off the VM. User initiated shutdown & reboots as well as power actions within vCenter do not impact the disposable disk.  The following actions will cause the disposable disk to be refreshed:

  • Rebalance
  • Refresh
  • Recompose
  • VM powered off due to the Pool Power Policy set to “Always Powered Off”

This is quite important to understand, as if the Pool Power Policy is set to any of the other settings (Powered On, Do Nothing or Suspend) then your disposable disks are not getting refreshed automatically.

What does all this mean?

Understanding Disposable Disks and their functionality will enable you to design your environment appropriately.  The View Storage Reclamation Feature that was introduced in View 5.2 uses an SE Sparse disk for the OS Disk, this allows View to shrink OS disks if files are deleted from within the OS.  However only the OS disk is created as an SE Sparse disk, User Data Disks and Disposable Disks are created as a standard VMDK.  The key difference with this feature compared with Disposable Disks, is it relies on files being deleted from within the Guest Operating System, where as the Disposable Disk is deleted along with all the files it contains when View powers off the VM.  It is also important to note, that currently SE Sparse disks are not supported on VSAN.

If you choose to use Disposable Disks in your design, then depending on your power cycle you may want to add an operational task for administrators to periodically change the Power On setting for the pool within a maintenance window to refresh the Disposable Disk.  This is particularly important for the use case of Persistent Desktops which have long refresh/recompose cycles.


Travis Wood is a VMware Senior Solutions Architect

“Gotchas” and Lessons Learned When Using Virtual SAN

jonathanm-profileBy Jonathan McDonald

There are certainly a number of blogs on the Web that talk about software-defined storage, and in particular Virtual SAN. But as someone who has worked at VMware for nine years, my goal is not to rehash the same information, but to provide insights from my experiences.

At VMware, much of my time was spent working for Global Support Services; however, over the last year-and-a-half, I have been working as a member of the Professional Services Engineering team.

As a part of this team, my focus is now on core virtualization elements, including vSphere, Virtual SAN, and Health Check Services. Most recently I was challenged with getting up to speed with Virtual SAN and developing an architecture design for it. At first this seemed pretty intimidating, since I had only heard about the marketing details prior to this; however, Virtual SAN truly did live up to all the hype about being “radically simple”. What I found is that the more I work with Virtual SAN the less concerned I became with the underlying storage. After having used Virtual SAN and tested it in customer environments, I can honestly say my mind is very much changed because of the absolute power it gives an administrator.

To help simplify the design process I broke it out into the following workflow design to not only simplify it for myself, but to help anyone else who is unaware of the different design decisions required to successfully implement Virtual SAN.

Workflow for a Virtual SAN Design_JMcDonald

Workflow for a Virtual SAN Design

When working with a Virtual SAN design, this workflow can be quite helpful. To further simplify it, I break it down into a four key areas:

  1. Hardware selection – In absolutely every environment I have worked in there has always been a challenge to select the hardware. I would guess that 75 percent of the problems I have seen in implementing Virtual SAN have been as a result of hardware selection or configuration. This includes things such as non-supported devices or incorrect firmware/drivers. Note: VMware does not provide support for devices that are not on the Virtual SAN Compatibility List. Be sure that when selecting hardware that it is on the list!
  2. Software configuration – The configuration is simple—rarely have I seen questions on actually turning it on. You merely click a check box, and it will configure itself (assuming of course that the underlying configuration is correct). If it is not, the result can be mixed, such as if the networking is not configured correctly, or if the disks have not been presented properly.
  3. Storage policy – The storage policy is at first a huge decision point. This is what gives Virtual SAN its power, the ability to configure what happens with the virtual machine for performance and availability characteristics.
  4. Monitoring/performance testing/failure testing – This is the final area and it is in regards to how you are supposed to monitor and test the configuration.

All of these things should be taken into account in any design for Virtual SAN, or the design is not really complete. Now, I could talk through a lot of this for hours. Rather than doing that I thought it would be better to post my top “gotcha” moments, along with the lessons learned from the projects I have been involved with.

Common “Gotchas”

Inevitably, “gotcha” moments will happen when implementing Virtual SAN. Here are the top moments I have run into:

  1. 1. Network configuration – No matter what the networking team says, always validate the configuration. The “Misconfiguration detected” error is by far the most common thing I have seen. Normally this means that either the port group has not been successfully configured for Virtual SAN or the multicast has not been set up properly. If I were to guess, most of the issues I have seen are as a result of multicast setup. On Cisco switches, unless an IGMP Snooping Carrier has been configured OR IGMP snooping has been explicitly disabled on the ports used for Virtual SAN, configuration will generally fail. In the default configuration it is simply not configured, and therefore—even if the network admin says it is configured properly it may not be configured at all—double check it to avoid any painNetwork Configuration_JMcDonald
  2. Network speed – Although 1 GB networking is supported, and I have seen it operate effectively for small environments, 10 GB networking is highly recommended for most configurations. I don’t just say this because the documentation says so. From experience, what it really comes down to here is not the regular everyday usage of Virtual SAN. Where people run into problems rather is when an issue occurs, such as during failures or periods of heavy virtual machine creation. Replication traffic during these periods can be substantial and cause huge performance degradation while they are occurring. The only way to know is to test what happens during a failure or peek provisioning cycle. This testing is critical as this tells you what the expected performance will be. When in doubt, always use 10 GB networking.
  3. Storage adapter choice – Although seemingly simple, the queue depth of the controller should be greater than 256 to ensure the best performance. This is not as much of an issue now as it was several months ago because the VMware Virtual SAN compatibility list should no longer have any cards that are under 256 queue depth in it anymore. Be sure to verify though. As an example, there was one card when first released that artificially limited the queue depth of the card in the driver software. Performance was dramatically impacted until an updated driver was released.

Lessons Learned

There are always lessons to be learned when using new software, and ours came with a price of a half or full day’s work in trying to troubleshoot issues. Here’s what we figured out:

  1. Always verify firmware/driver versions – This one always seems to be overlooked, but I am stating it because of experiences onsite with customers.One example that comes to mind is where we had three identical servers bought and shipped in the same order that we were using to configure Virtual SAN. Two of them worked fine, the third just wouldn’t cooperate, no matter what we did. After investigating for several hours we found that not only would Virtual SAN not configure, but all drives attached to that host were Read only. Looking at the utility that was provided with the actual card itself showed that the card was a revision behind on the firmware. As soon as we upgraded the firmware it came online and everything worked brilliantly.
  2. Pass-through/RAID0 controller configuration – It is almost always recommended to use a pass-through controller such as Virtual SAN, as it is the owner of the drives and can have full control of them. In many cases there is only RAID0 mode. Proper configuration of this is required to avoid any problems and to maximize performance for Virtual SAN. First, ensure any controller caching is set to 100% Read Cache. Second, configure each drive as its own “array” and not a giant array of disks. This will ensure it is set up properly.As an example of incorrect configuration that can cause unnecessary overhead, several times I have seen all disks configured as a single RAID volume on the controller. This shows up as a single disk to the operating system (ESXi in this case), which is not desired for Virtual SAN. To fix this you have to go into the controller and configure it correctly, by configuring each disk individually.  You also have to ensure the partition table (if previously created) is removed, which can—in many cases—involve a zero out of the drive if there is not an option to remove the header.
  3. Performance testing – The lesson learned here is you can do an infinite amount of testing – where do you start and stop. Wade Holmes from the Virtual SAN technical marketing team at VMware has an amazing blog series on this that I highly recommend reviewing for guidance here. His methodology allows for both basic and more in-depth testing to be done for your Virtual SAN configuration.

I hope these pointers help in your evaluation and implementation of Virtual SAN. Before diving head first in to anything, I always like to make sure I am informed about the subject matter. Virtual SAN is no different. To be successful you need to make sure you have genuine subject matter expertise for the design, whether it be in-house or by contacting a professional services organization. Remember, VMware is happy to be your trusted advisor if you need assistance with Virtual SAN or any of our other products!


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments. 

Creating Purpose-Built vSphere Clusters

By Sunny Dua, Senior Technology Consultant at VMware 

I recently had an opportunity to present at vForum 2013 in Mumbai, the Financial Capital of India. With more than 3,000 participants and two days of events, it was definitely one of the biggest customer events in India. Along with my team, I represented VMware Professional Services and presented on the following topic: “Architecting vSphere Environments – Everything you wanted to know!”

When we finalized the topic, I realized that presenting this topic in 45 minutes is next to impossible. With the amount of complexity that goes into Architecting a vSphere Environment, one could easily write an entire book. However, the task at hand was to keep it to the length of a presentation.

As I started planning the slides, I decided to look at the architectural decisions, which in my experience are the Most Important Ones, since they can make or break the virtual infrastructure. My other criterion was to ensure I talk about the Grey Areas where I always see uncertainty. This uncertainty can transform a good design into a bad one.

At the end I was able to come out with a final presentation which was received very well by the attendees. I thought of sharing the content with the entire community through this blog post. This is part 1, where I will give you some key design considerations for designing vSphere Clusters.

Before I begin, I also want to give the credit to a number of VMware experts in the community. Their books, blogs and the discussions I have had with them in the past helped me in creating this content. This includes books and blogs by DuncanFrankForbes GuthrieScott LoweCormac Hogan and some fantastic discussions with Michael Webster earlier this year.

Before we begin here is a small graphical disclaimer:

And here are my thoughts on creating vSphere Clusters.

The message behind the slide above is to create vSphere Clusters based on the purpose they need to fulfill in the IT landscape of your organization.

Management Cluster

The management cluster refers here to a 2- to 3-host ESXi host used by the IT team to primarily host all the workloads that are used to build up a vSphere Infrastructure. This includes VMs such as vCenter Server, Database Server, vCOps, SRM, vSphere Replication Appliance, VMA Appliance, Chargeback Manager, etc. This cluster can also host other infrastructure components such as active directory, backup servers, anti-virus etc. This approach has multiple benefits such as:

  • Security due to isolation of management workloads from production workloads. This gives complete control to the IT team on the workloads, which are critical to manage the environment.
  • Ease of upgrading the vSphere Environment and related components without impacting the production workloads.
  • Ease of troubleshooting issues within these components since the resources such as compute, storage, and network are isolated and dedicated for this cluster.
  • The number of ESXi hosts in a cluster will impact your consolidation ratios in most cases. As a rule of thumb, you will always consider one ESXi host in a 4-node cluster for HA failover (assuming), but you could also do the same on a 8-node cluster, which ideally saves one ESXi host for you for running additional workloads. Yes, the HA calculations matter and they can be either on the basis of slot size or percentage of resources.
  • Always consider at least one host as a failover limit per 8 to 10 ESXi servers. So in a 16 node cluster, do not stick with only one host for failover, look for at least taking this number to two. This is to ensure that you cover the risk as much as possible by providing an additional node for failover scenarios.
  • Setting up large clusters comes with its benefits, such as higher consolidation ratios etc. But they might have a downside as well if you do not have enterprise-class or rightly sized storage in your infrastructure. Remember, if a datastore is presented to a 16-node or a 32-node cluster, and if the VMs on that datastore are spread across the cluster, chances are you might get into contention for SCSI locking. If you are using VAAI, this will be reduced by ATS; however, try to start small and grow gradually to see if your storage behavior is not being impacted.
  •  Having separate ESXi servers for DMZ workloads is OLD SCHOOL. This was done to create physical boundaries between servers. This practice is a true burden carried over from the physical world to the virtual. It’s time to shed that load and make use of mature technologies, such as VLANs to create logical isolation zones between internal and external networks. Worst case, you might want to use separate network cards and physical network fabric, but you can still run on the same ESXi server, giving you better consolidation ratios and ensuring the level of security required in an enterprise.

Quick Tip: Ensure that this cluster is a minimum 2-node cluster for vSphere HA to protect workloads in case one host goes down. A 3-node management cluster would be ideal, since you would have the option of running maintenance tasks on ESXi servers without having to disable HA. You might want to consider using VSAN for this infrastructure as this is the primary use case that both Rawlinson & Cormac suggest. Remember, VSAN is in beta right now, so make your choices accordingly.

Production Clusters

As the name suggests this cluster would host all your production workloads. This cluster is the heart of your organization as it hosts the business applications, databases, and web services. This is what gives you the job of being a VMware architect or a virtualization admin.

Here are a few pointers to keep in mind while creating production clusters:

  • The number of ESXi hosts in a cluster will impact you consolidation ratios in most of the cases. As a rule of thumb, you will always consider one ESXi host in a 4-node cluster for HA failover (assuming), but you could also do the same on a 8-node cluster, which ideally saves one ESXi host for you for running additional workloads. Yes, the HA calculations matter and they can be either on the basis of slot size or percentage of resources.
  • Always consider at least 1 host as a failover limit per 8 to 10 ESXi servers. So in a 16 node cluster, do not stick with only 1 host for failover, look for at least taking this number to 2. This is to ensure that you cover the risk as much as possible by providing additional node for failover scenarios
  • Setting up large clusters comes with their benefits such as higher consolidation ratios etc., they might have a downside as well if you do not have the enterprise class or rightly sized storage in your infrastructure. Remember, if a Datastore is presented to a 16 Node or a 32 Node cluster, and on top of that, if the VMs on that datastore are spread across the cluster, chances that you might get into contention for SCSI locking. If you are using VAAI this will be reduced by ATS, however try to start with small and grow gradually to see if your storage behavior is not being impacted.

Having separate ESXI servers for DMZ workloads is OLD SCHOOL. This was done to create physical boundaries between servers. This practice is a true burden which is carried over from physical world to virtual. It’s time to shed that load and make use of mature technologies such as VLANs to create logical isolation zones between internal and external networks. In worst case, you might want to use separate network cards and physical network fabric but you can still run on the same ESXi server which gives you better consolidation ratios and ensures the level of security which is required in an enterprise.

Island Clusters

They sound fancy but the concept of island clusters is fairly simple: run islands of ESXi servers (small groups) that can host workloads with special license requirements. Although I do not appreciate how some vendors try to apply illogical licensing policies on their applications, middle-ware and databases, this is a great way of avoiding all the hustle and bustle created by sales folks. Some examples of island clusters would include:

  • Running Oracle Databases/Middleware/Applications on their dedicated clusters. This will not only ensure that you are able to consolidate more and more on a small cluster of ESXi hosts and save money but also ensures that you zip the mouth of your friendly sales guy by being in what they think is license compliance.
  • I have customers who have used island clusters of operating systems such as Windows. This also helps you save on those datacenter, enterprise, or standard editions of Windows OS.
  • Another important benefit of this approach is that it helps ESXi use the memory management technique of Transparent Page Sharing (TPS) more efficiently since there are chances that you are running a lot of duplicate pages spawned by these VMs in the physical memory of your ESXi servers. I have seen this reach 30 percent and can be fetched in a vCenter Operations Manager report if you have that installed in your virtual infrastructure.

With this I would close this article. I was hoping to give you a quick scoop in all these parts, but this article is now four pages. I hope this helps you make the right choices for your virtual infrastructure when it comes to vSphere Clusters.


This post originally appeared on Sunny Dua’s vXpress blog, where you can find follow-up posts 2 and 3. Sunny Dua is a Senior Technology Consultant for VMware’s Professional Services Organization, focused on India and SAARC countries.