Home > Blogs > VMware Consulting Blog > Tag Archives: VMware Professional Services

Tag Archives: VMware Professional Services

“Gotchas” and Lessons Learned When Using Virtual SAN

jonathanm-profileBy Jonathan McDonald

There are certainly a number of blogs on the Web that talk about software-defined storage, and in particular Virtual SAN. But as someone who has worked at VMware for nine years, my goal is not to rehash the same information, but to provide insights from my experiences.

At VMware, much of my time was spent working for Global Support Services; however, over the last year-and-a-half, I have been working as a member of the Professional Services Engineering team.

As a part of this team, my focus is now on core virtualization elements, including vSphere, Virtual SAN, and Health Check Services. Most recently I was challenged with getting up to speed with Virtual SAN and developing an architecture design for it. At first this seemed pretty intimidating, since I had only heard about the marketing details prior to this; however, Virtual SAN truly did live up to all the hype about being “radically simple”. What I found is that the more I work with Virtual SAN the less concerned I became with the underlying storage. After having used Virtual SAN and tested it in customer environments, I can honestly say my mind is very much changed because of the absolute power it gives an administrator.

To help simplify the design process I broke it out into the following workflow design to not only simplify it for myself, but to help anyone else who is unaware of the different design decisions required to successfully implement Virtual SAN.

Workflow for a Virtual SAN Design_JMcDonald

Workflow for a Virtual SAN Design

When working with a Virtual SAN design, this workflow can be quite helpful. To further simplify it, I break it down into a four key areas:

  1. Hardware selection – In absolutely every environment I have worked in there has always been a challenge to select the hardware. I would guess that 75 percent of the problems I have seen in implementing Virtual SAN have been as a result of hardware selection or configuration. This includes things such as non-supported devices or incorrect firmware/drivers. Note: VMware does not provide support for devices that are not on the Virtual SAN Compatibility List. Be sure that when selecting hardware that it is on the list!
  2. Software configuration – The configuration is simple—rarely have I seen questions on actually turning it on. You merely click a check box, and it will configure itself (assuming of course that the underlying configuration is correct). If it is not, the result can be mixed, such as if the networking is not configured correctly, or if the disks have not been presented properly.
  3. Storage policy – The storage policy is at first a huge decision point. This is what gives Virtual SAN its power, the ability to configure what happens with the virtual machine for performance and availability characteristics.
  4. Monitoring/performance testing/failure testing – This is the final area and it is in regards to how you are supposed to monitor and test the configuration.

All of these things should be taken into account in any design for Virtual SAN, or the design is not really complete. Now, I could talk through a lot of this for hours. Rather than doing that I thought it would be better to post my top “gotcha” moments, along with the lessons learned from the projects I have been involved with.

Common “Gotchas”

Inevitably, “gotcha” moments will happen when implementing Virtual SAN. Here are the top moments I have run into:

  1. 1. Network configuration – No matter what the networking team says, always validate the configuration. The “Misconfiguration detected” error is by far the most common thing I have seen. Normally this means that either the port group has not been successfully configured for Virtual SAN or the multicast has not been set up properly. If I were to guess, most of the issues I have seen are as a result of multicast setup. On Cisco switches, unless an IGMP Snooping Carrier has been configured OR IGMP snooping has been explicitly disabled on the ports used for Virtual SAN, configuration will generally fail. In the default configuration it is simply not configured, and therefore—even if the network admin says it is configured properly it may not be configured at all—double check it to avoid any painNetwork Configuration_JMcDonald
  2. Network speed – Although 1 GB networking is supported, and I have seen it operate effectively for small environments, 10 GB networking is highly recommended for most configurations. I don’t just say this because the documentation says so. From experience, what it really comes down to here is not the regular everyday usage of Virtual SAN. Where people run into problems rather is when an issue occurs, such as during failures or periods of heavy virtual machine creation. Replication traffic during these periods can be substantial and cause huge performance degradation while they are occurring. The only way to know is to test what happens during a failure or peek provisioning cycle. This testing is critical as this tells you what the expected performance will be. When in doubt, always use 10 GB networking.
  3. Storage adapter choice – Although seemingly simple, the queue depth of the controller should be greater than 256 to ensure the best performance. This is not as much of an issue now as it was several months ago because the VMware Virtual SAN compatibility list should no longer have any cards that are under 256 queue depth in it anymore. Be sure to verify though. As an example, there was one card when first released that artificially limited the queue depth of the card in the driver software. Performance was dramatically impacted until an updated driver was released.

Lessons Learned

There are always lessons to be learned when using new software, and ours came with a price of a half or full day’s work in trying to troubleshoot issues. Here’s what we figured out:

  1. Always verify firmware/driver versions – This one always seems to be overlooked, but I am stating it because of experiences onsite with customers.One example that comes to mind is where we had three identical servers bought and shipped in the same order that we were using to configure Virtual SAN. Two of them worked fine, the third just wouldn’t cooperate, no matter what we did. After investigating for several hours we found that not only would Virtual SAN not configure, but all drives attached to that host were Read only. Looking at the utility that was provided with the actual card itself showed that the card was a revision behind on the firmware. As soon as we upgraded the firmware it came online and everything worked brilliantly.
  2. Pass-through/RAID0 controller configuration – It is almost always recommended to use a pass-through controller such as Virtual SAN, as it is the owner of the drives and can have full control of them. In many cases there is only RAID0 mode. Proper configuration of this is required to avoid any problems and to maximize performance for Virtual SAN. First, ensure any controller caching is set to 100% Read Cache. Second, configure each drive as its own “array” and not a giant array of disks. This will ensure it is set up properly.As an example of incorrect configuration that can cause unnecessary overhead, several times I have seen all disks configured as a single RAID volume on the controller. This shows up as a single disk to the operating system (ESXi in this case), which is not desired for Virtual SAN. To fix this you have to go into the controller and configure it correctly, by configuring each disk individually.  You also have to ensure the partition table (if previously created) is removed, which can—in many cases—involve a zero out of the drive if there is not an option to remove the header.
  3. Performance testing – The lesson learned here is you can do an infinite amount of testing – where do you start and stop. Wade Holmes from the Virtual SAN technical marketing team at VMware has an amazing blog series on this that I highly recommend reviewing for guidance here. His methodology allows for both basic and more in-depth testing to be done for your Virtual SAN configuration.

I hope these pointers help in your evaluation and implementation of Virtual SAN. Before diving head first in to anything, I always like to make sure I am informed about the subject matter. Virtual SAN is no different. To be successful you need to make sure you have genuine subject matter expertise for the design, whether it be in-house or by contacting a professional services organization. Remember, VMware is happy to be your trusted advisor if you need assistance with Virtual SAN or any of our other products!


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments. 

Holistic Engagements Lead to Successful Outcomes

Ford DonaldBy Ford Donald, Principal Architect, GTS PSE, VMware

In my last post, I introduced an optimized consulting approach called the SDDC Assess, Design, and Deploy Service. The post focused on the technical blueprint, designed with common core elements, and the flexibility for custom implementation using modular elements. In this post, we’ll explore the process improvements that lead to holistic, mutually beneficial engagements.

The Work Stream Process
The six-step process takes into account both our prescribed starting point—the technical foundation—and the unique needs of the customer, with an eye towards a predictable outcome.

1. Solution Overview. We begin with an overview of the technical foundations and the new approach to help the customer understand the benefits of holistic consultation and the specific solution design. This sets a level discussion between the modeled approach and the pre-conceptions of how things work. Stepping back to review the approach gets us to the assessment phase quickly so we are all on the same page about how we’ll be working together.

2. Assessment Phase. In this phase, we assess what the customer already has in place, and where they would like to be at the end of the project. Some customers have strong opinions of design, others don’t. Defined gaps are where we come in with adaptations to the prescribed design, with layers and snap-ins added as desired.

3. Design Phase. Here, we bring forward the adapted solution, shaped to meet the customer’s needs and requirements, relative to our good starting point with the prescribed solution.

4. Deploy Phase. Given all the up-front work up to this point, deployment should be straightforward. We add what’s missing, modify what’s not right, and bulk up or whittle down to get to the adapted solution. Here we would add in things like Orchestrator if it’s not currently deployed, along with the Orchestration workflow library. These pre-defined, generalized, well-documented workflows are field-tested and designed so that we can easily provide support—this ensure that they are consistent across the board.

5. Knowledge Transfer. I like to call this the cool-down period. Here we take two steps back and let the environment learn, stabilize, and cool off a bit. For example, VCOps does best if it’s given three or four weeks to understand what normal is. This is a great time to train administrative staff on the new implementation and announce any operational or organizational transformations needed. It’s important to take the time to get a feeling for what’s new or changed, from interfaces and APIs to dealing with resources and loading up templates.

6. Solution Validation. In this phase we come together to look back and compare the results to the prescribed beginnings. If we haven’t hit the mark, remediation is required.

The Project Timeline
It’s important to note that each phase of the technical transformation has its own work stream process. No engagement should take on the entire thing as one major project. Rather, it should be a series of engagements that meet the customer’s timeline and adoption capability. The various stages will take place over a lengthy time period.

Traditionally, customer engagements have focused on the assessment or the design and deliver phase. By adding in the Solution Overview, and ensuring we’re all starting from the same point, we lay the foundation for success.


Ford Donald is a Principal Architect and member of Professional Services Engineering (PSE), a part of the Global Technical Solutions (GTS) team, a seven-year veteran of VMware. Prior to PSE, Ford spent three years as a pre-sales cloud computing specialist focusing on very large/complex virtualization deployments, including the VMware sales cloud known as vSEL. Ford also served as coreteam on VMworld Labs and as a field SE.

New Technology Implementation Plan: Start by Stepping Back

Jeremy Carter headshotBy Jeremy Carter, VMware Senior Consultant

I’ve been working on a customer engagement recently that takes advantage of vCloud Automation Center (vCAC), which is designed to centralize and automate key IT activities, freeing the organization to focus on the needs of internal and external customers.

In our deployment of vCAC, I’ve been reminded of a key principal of IT and business transformation: The technology is only part of the process. Often a shift in technology requires a period of assessment and realignment that is as valuable as the technology itself.

When the VMware Professional Services team is brought in for an engagement, the company wants to get the best return on its investment, so the IT team is receptive to our schedule of meetings and stock-taking. But every IT organization will benefit by starting their new technology implementation plan by stepping back to survey the systems in place before integrating a new one.

We put a lot of emphasis on investigating how things are currently done, often starting by asking the teams to draw their processes, for creating a virtual machine, for instance. Frequently we find they have two or three different processes in place, depending on who’s making request. This is especially common in government and higher education, where each department is likely to have it’s own IT team and strategy.

The unfortunate fact is that automation still scares people, thinking they’re going to be out of a job. On the contrary, if you look at any IT organization out there, you’ll see that it’s overwhelmed with tasks, many of which are never getting done. Automation can give them time back to focus on what’s important to their customers.

A new implementation is a perfect opportunity to look at which processes are working the best and align all the teams to them. When a team sees that they’ll be able to provide a better experience and quicker turnaround, their resistance to automation often fades.

And luckily vCAC provides enough flexibility that users don’t have to adopt exactly the same systems across the organization. With a college I worked with recently, we were able to build on what teams are already doing. Next we focused on handoff systems to cut down on the number of emails flying around: one for DNS, another to install the OS, etc.

This process—of assessing current processes, building in automation and consistency, and then refocusing on customer needs—is undeniably valuable. But it does take time. It’s worth putting these reassessments on the calendar every 6 or 12 months; if that doesn’t work, I recommend taking the opportunity presented by the implementation of a new technology to keep moving toward the best your organization can be.


Jeremy Carter is a Senior Consultant with VMware and is focused on the Software Defined Data Center (SDDC). He has special expertise in cloud infrastructure and automation, and BCDR. Over his 14 years in IT he has gained a variety of experience as an architect, DBA, and developer. Prior to joining VMware, Jeremy was a Principal Architect at one of the largest VMware service providers. 

 

4 Key Steps for Successful Infrastructure Implementation

By Martijn Baecke, VMware Senior Consultant

As a follow-up to the infographic Bret Connor posted last week about the ways VMware Professional Services collaborate with clients, I thought I would share some tips from my experience working with a client. Whether you’re leading your own IT engagement or working internally, I hope this will help you start build a strong foundation for your next implementation.

The Client

This engagement was with a European ministry providing IT services to five government agencies, and needing to extend its reach to eleven. Its goal was to implement a single infrastructure able to deliver shared services where needed; however, its two data centers were already approaching capacity.

Our charter was to design the infrastructure for the ministry’s current needs—consolidating agencies into a single IT platform—while also developing a roadmap for migration to a cloud architecture in two years. Acting on the advice of VMware, the ministry decided to replace its aging hardware with blade servers, upgrade to the latest version of vSphere, and virtualize all major business applications.

1. Discovery

To understand where you want to go, it’s important to understand where you’re starting from. That’s why one of our first steps was to document in detail the ministry’s current architecture, along with business requirements, technical constraints, and other design parameters.

From there we were able to narrow in on a few key goals:

  1. Simplify data center management
  2. Automate important processes
  3. Improve resiliency
  4. Respond faster to shifting priorities

2. Research & Buy-in

Early on, we hosted several workshops to determine needs and characteristics according to stakeholders at every level—users, managers, directors, and above. Making sure to gather input from a broad cross section helps avoid late-stage direction shifts, and also helps gain buy-in for the chosen solution.

For more about gaining buy-in from the highest levels and finding someone to champion your cause, I recommend Samuel Denton-Giles’ excellent post from December.

3. Planning

Considering the goals of the IT organization as well as other ministry departments, we were able to help them plan both a near-term refresh and a longer-term roadmap to the cloud. The most significant high-level recommendation was to adopt a building-block architecture: a modular system sized to fit existing needs that could easily scale to match future demand.

4. Education & Hand-off

To help avoid vision drift after hand-off, we were careful to map each requirement that came up in the initial forum to a specific technology we helped put in place to support it. Our consulting team also shared best practices and technology standards with the ministry’s IT staff in presentations and informal discussions.

At the end of the engagement, the ministry’s IT managers had a much clearer understanding of how the cloud would impact day-to-day operations, from help desk operations and staff scheduling to management and training. Ultimately this helped the ministry’s IT staff approach its future cloud expansion with confidence, knowing they would avoid expensive, disruptive missteps.


Martijn Baecke is a Senior Consultant for VMware Professional Services in Northern EMEA. He has 10+ years experience in advising and consulting with large enterprise companies around IT infrastructure. He is a VMware Certified Design eXpert (VCDX #103) and you can find more insights on his personal blog, Think©Loud.

Inside VMware Professional Services – Connect, Empower, Focus

From Bret Connor

Happy New Year! As vice president of VMware Professional Services for the Americas delivery team, I am fortunate to work with and lead a team of gifted, creative and downright smart professionals. How do we help our customers? Simple—we deliver faster results while cutting costs and accelerating business breakthroughs that have a material impact on your top line. I can go on but instead, we produced a visual guide that tells the story of the three cornerstone principles that differentiate our work from our competitors. Enjoy!