Home > Blogs > VMware Consulting Blog > Monthly Archives: May 2016

Monthly Archives: May 2016

Demo – Dynamically Enforcing Security on a Hot Cloned SQL Server with VMware NSX

Originally posted on the Virtualize Business Critical Applications blog.

Niran_Even_Chen

 

By Niran Even-Chen

VMware NSX is a software defined solution that brings the power of virtualization to network and security.

There are many great papers about NSX in general: for example here & here and many others, the purpose of this demo is not to dive into everything that NSX does. Instead I have focused on one capability in particular and that is the intelligent grouping of NSX Service Composer with the Distributed VMware NSXFirewall (DFW) and how to utilize it to make life easier for SQL DBAs and security admins, its doesn’t have to be only SQL Server, it can be any other database or application for that matter but for this demo I am focusing on SQL Server.

First, a bit of background: The NSX Service Composer allows us to create groups called “Security groups”. These Security groups can have a dynamic membership criteria that can be based on multiple factors: It can be part of the computer name of a VM, its guest OS name, the VM name, AD membership or a tag (tags are especially cool as they can be set automatically by 3rd party tools like antivirus and IPSs, but that is for a different demo)

These Security groups are than placed inside the Distributed Firewall (DFW) rules which allows us to manage thousands of entities with just a few rules and without the need to add these entities to the Security Group manually.

In the demo I have created an environment that is set with 0 trust policy, that means that everything is secured and every packet between the VMs is inspected, the inspection is done on the VMs vNIC level in an east-west micro segmentation way. That means that if a certain traffic is not defined in the DFW it is not allowed to go through.

This is something that wasn’t really possible to do before NSX

Our production app database is an SQL database and in the demo the DBA wants to hot-clone it aside for testing purposes, but obviously the cloned SQL Server needs to have some network traffic allowed to pass to it, yet it needs to be secured from everything else.

Instead of having the traditional testing FW zone with its own physical servers, I created the rules that apply to a test DBs in the DFW, created a dynamic membership Security Group, and nested that group in the rules. Now, any database server that I will clone which corresponds to the criteria will be automatically placed in the rules.  What’s really nice about this is that no traffic is going northbound to the perimeter FW because the packet inspection is done on the vNIC of the VMs (and only relevant rules to it are set on it) , no additional calls to security admins to configure the FW are needed after the first configuration has been made. This is a huge time saver, much more efficient in terms of resources (physical servers are now shared between zones) and a much more secure environment than having only a perimeter FW.

As usual, any comment or feedback is welcome

Cheers,

Niran


Niran is a VMware Staff Solutions Architect in the Enterprise Application Architecture team at who is focused on creating solutions for running Microsoft OS’s and apps on vSphere and vCloud Air platforms and providing top deal support to strategic customers globally.

Define SDDC Success Based on IT Outcomes

Andrea SivieroBy Andrea Siviero

You’ve just deployed a new technology solution; how do you define whether or not it was a success?

People often have difficulty agreeing on the definition of “success” because there are two interconnected dimensions in which a project can be judged as a success or a failure. The first is project management success (delivering in accordance with the agreed-upon project objectives), and the second is 0utcome success (the amount of value the project delivers once it is complete).

Of course, getting agreement on how to define success is not always easy, but based on my day-to-day experience with customers, outcome success is desired over project management success.

Outcomes Are Worth More Than Services

Buying a service rather than an outcome is similar to paying to use equipment at a gym versus working with a personal trainer, whose job is to help you produce an outcome. The latter is worth more than the former.

VMware’s IT Outcomes support the top priority initiatives for CIOs and impact key business metrics, you can check the dedicated web site here.

In my (humble) opinion, indifferently by the IT Outcomes you are focus on, there are three important factors that contribute to a success:

People, Processes, and Architecture.

Based on my experience, customers tend to focus on architecture and technology, sometimes paying less attention to the people and process factors which can contribute more to success. Here is a real-life example from my personal experience.

ASiviero_Simplify the Situation

I was involved with a successful project implementation where all the project’s technical objectives were achieved, but the infrastructure and operations manager did not feel the desired outcomes were achieved. And that manager was right!

After spending an hour talking with the teams, I realized what a great job the consultants had done implementing and demonstrating all the capabilities of their new SDDC.

However, due to their experience, expectations, and culture, they weren’t able to reorganize their teams and processes to take full advantage of the desired outcomes (Speed, Agility and Security).

ASiviero_Amazing SDDC

Here is a summary of the best practices I’ve suggested as a way to leverage VMware technical account managers as coaches.

1 – People

ASiviero_Small Cross Functional Team

  1. Create a blended team of skilled workers with multi-domain and multi-disciplinary knowledge and expertise, and deliver cross-team training.
  1. Encourage autonomy with common goals and operating principles, and focus on service delivery.
  1. Push them to share lessons learned with other teams and expand their use of virtual networking and security.

2 – Process

ASiviero_Application Level Visibility

  1. Decompose management and troubleshooting tasks along virtual and physical boundaries.
  1. Automate manual tasks to improve efficiency and reduce errors.
  1. Correlate the end-to-end view of application health across compute, storage, and networking.

3 – Architecture

ASiviero_Key Requirements for SDDC

  1. Build your SDDC using a design validated by experts.
  1. Implement a comprehensive data center design.
  1. Add in app and network virtualization incrementally.

Putting it all together

ASiviero_Putting it All Together

Achieving 100% of a project’s intended outcomes depends not only on the technology implementation, but also on the organizatonal transformation required to ensure the proper implementation of people and process innovation.


Andrea Siviero is an ten-year veteran of VMware and a senior solutions architect member of Professional Services Engineering (PSE) for the Software-Defined Datacenter (SDDC), a part of the Global Technical Solutions (GTS) team. Prior to PSE, Andrea spent three years as pre-sales system engineer and three years as a post-sales consultant architect for cloud computing and desktop virtualization solutions focusing on very large and complex deployments, especially for service providers in the finance and telco sectors.

Troubleshooting Tips: Orchestrator PowerShell Plug-in

By Spas Kaloferov

Background and General Considerations

In this post will we will take a look at some common issues one might experience when using the VMware vRealize Orchestrator (vRO) PowerShell Plug-In, especially when using HTTPS protocol or Kerberos authentication for the PowerShell Host (PSHost).

Most use cases require that the PowerShell script run with some kind of administrator-level permissions in the target system that vRO integrates with. Here are some of them:

  • Add, modify, or remove DNS records for virtual machines.
  • Register IP address for a virtual machine in an IP management system.
  • Create, modify, or remove a user account mailbox.
  • Execute remote PowerShell commands against multiple Microsoft Windows operating systems in the environment.
  • Run a PowerShell script (.ps1) file from within a PowerShell script file from vRO.
  • Access mapped network drives from vRO.
  • Interact with Windows operating systems that have User Access Control (UAC) enabled.
  • Execute PowerCLI commands.
  • Integrate with Azure.

When you add a PowerShell Host, you must specify a user account. That account will be used to execute all PowerShell scripts from vRO. In most use cases, like the one above, that account must be an administrator account in the corresponding target system the script interacts with. In most cases, this is a domain-level account.

In order to successfully add the PowerShell Host to that account—and use that account when executing scripts from vRO—some prerequisites need to be met. In addition, the use cases mentioned require the PowerShell Host to be prepared for credential delegation (AKA Credential Security Service Provider [CredSSP], double-hop authentication or multi-hop authentication).

To satisfy the above use cases for adding a PowerShell Host in vRO:

The high-level requirements are:

  • Port: 5986
  • PowerShell remote host type: WinRM
  • Transport protocol: HTTPS (recommended)
  • Authentication: Kerberos
  • User name: <Administrator_user_name>

The low-level requirements are:

  • PSHost: Configure WinRM and user token delegation
  • PSHost: Configure Windows service principal names (SPNs) for WinRM
  • PSHost: Import a CA signed-server certificate containing Client Authentication and Server authentication Exchange Key Usage Properties
  • PSHost: Configure Windows Credential Delegation using the Credential Security Service Provider (CredSSP) module
  • vRO: Edit the Kerberos Domain Realm (krb5.conf) on the vCO Appliance (Optional/Scenario specific)
  • vRO: Add the PS Host as HTTPS host with Kerberos authentication
  • vRO: Use the Invoke-Command cmdlet in your PowerShell code

Troubleshooting Issues when Adding a PSHost

To resolve most common issues when adding a PSHost for use with HTTPS transport protocol and Kerberos authentication, follow these steps:

  1. Prepare the Windows PSHost.

For more information on all the configurations needed on the PSHost, visit my blog, “Using CredSSP with the vCO PowerShell Plug-in.”

  1. After preparing the PSHost, test it to make sure it accepts the execution or removes PowerShell commands.

Start by testing simple commands. I like to use the $env:computername PowerShell command that returns the hostname of the PSHost. You can use the winrs command in Windows for the test. Here’s an example of the syntax:

winrs -r:https://lan1dc1.vmware.com:5986 -u:vmware\administrator -p:VMware1! powershell.exe $env:computername

 

Continue by testing a command that requires credential delegation. I like to use a simple command, like dir \\<Server_FQDN\<sharename>, that accesses a share residing on a computer other than the PSHost itself. Here’s an example of the syntax:

winrs -r:https://lan1dc1.vmware.com:5986 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share


Note
: Make sure to specify the –ad command line switch.

  1. Prepare the vRO so it can handle Kerberos authentication. You need this in order to use a domain-level account when adding the PSHost.

For more information about the Kerberos configuration on vRO for single domain, visit my blog, “Using CredSSP with the vCO PowerShell Plugin.”

If you are planning to add multiple PSHosts and are using domain-level accounts for each PSHost that are from different domains (e.g., vmware.com and support.vmware.com) you need to take this into consideration when preparing vRO for Kerberos authentication.

For more information about the Kerberos configuration on vRO for multiple domains, visit my blog, “How to add PowerShell hosts from multiple domains with Kerberos authentication to the same vRO.”

If you make a mistake in the configuration, you might see the following error then adding the PSHost:

Cannot locate default realm (Dynamic Script Module name : addPowerShellHost#12
tem: ‘Add a PowerShell host/item8′, state: ‘failed’, business state: ‘Error’, exception: ‘InternalError: java.net.ConnectException: Connection refused (Workflow:Import a certificate from URL with certificate alias / Validate (item1)#5)’
workflow: ‘Add a PowerShell

 

If this is the case, go back and re-validate the configurations.

  1. If the error persists, make sure the conf file is correctly formatted.

For more information about common formatting mistakes, visit my blog, “Wrong encoding or formatting of Linux configuration files can cause problems in VMware Appliances.”

  1. Make sure you use the following parameters when adding the PSHost:
    • Port: 5986
    • PowerShell remote host type: WinRM
    • Transport protocol: HTTPS (recommended)
    • Authentication: Kerberos
    • User name: <Administrator_user_name>

Note: In order to add the PSHost, the user must be a local administrator on the PSHost.

  1. If you still cannot add the host, make sure your VMware appliance can authenticate successfully using Kerberos against the domains you’ve configured. To do this you can use the ldapsearch command and test Kerberos connectivity to the domain.

Here is an example of the syntax:

vco-a-01:/opt/vmware/bin # ldapsearch -h lan1dc1.vmware.com -D “CN=Administrator,CN=Users,DC=vmware,DC=com” -w VMware1! -b “” -s base “objectclass=*”
  1. If your authentication problems continue, most likely there is a general authentication problem that might not be directly connected to the vRO appliance, such as:
    • A network related issue
    • Blocked firewall ports
    • DNS resolution problems
    • Unresponsive domain controllers

Troubleshooting Issues when Executing Scripts

Once you’ve successfully added the PSHost, it’s time to test PowerShell execution from the vRO.

To resolve the most common issues when executing PowerShell scripts from vRO, follow these steps:

  1. While in vRO go to the Inventory tab and make sure you don’t see the word “unusable” in front of the PSHost name. If you do, remove the PSHost and add it to the vRO again.
  1. Use the Invoke an external script workflow that is shipped with vRO to test PowerShell execution commands. Again, start with a simple command, like $env:computername.

Then, process with a command that requires credential delegation. Again, as before, you can use a command like dir \\<Server_FQDN\<sharename>.

Note: This command doesn’t support credential delegation, so a slight workaround is needed to achieve this functionality. You need to wrap the command you want to execute around an Invoke-Command command.

For more information on how to achieve credential delegation from vRO, visit my blog, “Using CredSSP with the vCO PowerShell Plug-in.”

If you try to execute a command that requires credential delegation without using a workaround, you will receive an error similar to the following:

PowerShellInvocationError: Errors found while executing script <script>: Access is denied


SKaloferov_Power Shell Error

  1. Use the SilentlyContinue PowerShell error action preference to suppress output from “noisy” commands. Such commands are those that generate some kind of non-standard output, like:
    • Progress par showing the progress of the command execution
    • Hashes and other similar content

Finally, avoid using code in your commands or scripts that might generate popup messages, open other windows, or open other graphical user interfaces.


Spas Kaloferov is an acting Solutions Architect member of Professional Services Engineering (PSE) for the Software-Defined Datacenter (SDDC) – a part of the Global Technical & Professional Solutions (GTPS) team. Prior to VMware, Kaloferov focused on cloud computing solutions.

Virtualization and VMware Virtual SAN … the Old Married Couple

Don’t Mistake These Hyper-Converged Infrastructure Technologies as Mutually Exclusive

Jonathan McDonaldBy Jonathan McDonald

I have not posted many blogs recently as I’ve been in South Africa. I have however been hard at work on the latest release of VMware vSphere 6.0 Update 2 and VMware Virtual SAN 6.2. Some amazing features are included that will make life a lot easier and add some exciting new functionality to your hyper-converged infrastructure. I will not get into these features in this post, because I want to talk about one of the bigger non-technical questions that I get from customers and consultants alike. This is not one that is directly tied to the technology or architecture of the products. It is the idea that you can go into an environment and just do Virtual SAN, which from my experience is not true. I would love to know if your thoughts and experiences have shown you the same thing.

Let me first tell those of you who are unaware of Virtual SAN that I am not going to go into great depth about the technology. The key is that, as a platform, it is hyper-converged, meaning it is included with the ESXi hypervisor. This makes it radically simple to actually configure—and, more importantly, use—once it is up and running.

My hypothesis is that 80 to 90% of what you have to do to design for Virtual SAN focuses on the Virtualization design, and not so much on Virtual SAN.  This is not to say the Virtual SAN design is not important, but virtualization has to be integral to the design when you are building for it. To prove this, take a look at what the standard tasks are when creating the design for the environment:

  1. Hardware selection, racking, configuration of the physical hosts
  2. Selection and configuration of the physical network
  3. Software installation of the VMware ESXi hosts and VMware vCenter server
  4. Configuration of the ESXi hosts
    • Networking (For management traffic, and for VMware vSphere vMotion, at a minimum)
    • Disks
    • Features (VMware vSphere High Availability, VMware vSphere Distributed Resource Scheduler, VMware vSphere vMotion, at a minimum)
  5. Validation and testing of the configuration

If I add the Virtual SAN-specific tasks in, you have a holistic view of what is required in most greenfield configurations:

  1. Configuration of the Virtual SAN network
  2. Turning on Virtual SAN
  3. Creating new policies (optional, as the default is in place once configured)
  4. Testing Virtual SAN

As you can see, my first point shows that the majority of the work is actually virtualization and not Virtual SAN. In fact, as I write this, I am even more convinced of my hypothesis. The first three tasks alone are really the heavy hitters for time spent. As a consultant or architect, you need to focus on these tasks more than anything. Notice above where I mention “configure” in regards to Virtual SAN, and not installation; this is because it is already a hyper-converged element installed with ESXi. Once you get the environment up and running with ESXi hosts installed, Virtual SAN needs no further installation, simply configuration. You turn it on with a simple wizard, and, as long as you have focused on the supportability of the hardware and the underlying design, you will be up and running quickly. Virtual SAN is that easy.

Many of the arguments I get are interesting as well. Some of my favorites include:

  • “The customer has already selected hardware.”
  • “I don’t care about hardware.”
  • “Let’s just assume that the hardware is there.”
  • “They will be using existing hardware.”

My response is always that you should care a great deal about the hardware. In fact, this is by far the most important part of a Virtual SAN engagement. With Virtual SAN, if the hardware is not on the VMware compatibility list, then it is not supported. By not caring about hardware, you risk data loss and the loss of all VMware support.

If the hardware is already chosen, you should ensure that the hardware being proposed, added, or assumed as in place is proper. Get the bill of materials or the quote, and go over it line-by-line if that’s what’s needed to ensure that it is all supported.

Although the hardware selection is slightly stricter than with an average design, it is much the same as any traditional virtualization engagement in how you come to the situation. Virtual SAN Ready nodes are a great approach and make this much quicker and simpler, as they offer a variety of pre-configured hardware to meet the needs of Virtual SAN. Along with the Virtual SAN TCO Calculator it makes the painful process of hardware selection a lot easier.

Another argument I hear is “If I am just doing Virtual SAN, that is not enough time.” Yes, it is. It really, really is. I have been a part of multiple engagements for which the first five tasks above are already completely done. All we have to do is come in and turn on Virtual SAN. In Virtual SAN 6.2, this is made really easy with the new wizard:

JMcDonald_Configure VSAN

Even with the inevitable network issues (not lying here; every single time there is a problem with networking), environmental validation, performance testing, failure testing, testing virtual machine creation workflows, I have never seen it take more than a week to do this piece for a single cluster regardless of size of configuration. In many cases, after three days, everything is up and running and it is purely customer validation that is taking place. As a consultant or architect, don’t be afraid of the questions customers ask in regards to performance and failures. Virtual SAN provides mechanisms to easily test the environment as well as see as what “normal” is.

Here are two other arguments I hear frequently:

  • “We have never done this before.”
  • “We don’t have the skillset.”

These claims are probably not 100% accurate. If you have used VMware, or you are a VMware administrator, you are probably aware of the majority of what you have to do here. For Virtual SAN, specifically, this is where the knowledge needs to be grown. I suggest a training, or a review of VMworld presentations for Virtual SAN, to get familiar with this piece of technology and its related terminology. VMware offers training that will get you up to speed on hyper-converged infrastructure technologies, and the new features of VMware vSphere 6.0 Update Manager 2 and Virtual SAN 6.2.

For more information about free learnings, check out the courses below:

In addition, most of the best practices you will see are not unfamiliar since they are vCenter- or ESXi-related. Virtual SAN Health gives an amazing overview that is frequently refreshed, so any issues you may be seeing are reported here; this also takes a lot of the guess work out of the configuration tasks as you can see from the screenshot below, as many, if not all of, the common misconfigurations are shown.

JMcDonald_VSAN Health

In any case, I hope I have made the argument that Virtual SAN is mostly a virtualization design that just doesn’t use traditional SANs for storage.  Hyper-converged infrastructure is truly bringing change to many customers. This is, of course, just my opinion, and I will let you judge for yourself.

Virtual SAN has quickly become one of my favorite new technologies that I have worked with in my time at VMware, and I am definitely passionate about people using it to change the way they do business. I hope this helps in any engagements that you are planning as well as to prioritize and give a new perspective to how infrastructure is being designed.


Jonathan McDonald is a Technical Solutions Architect for the Professional Services Engineering team. He currently specializes in developing architecture designs for core Virtualization, and Software-Defined Storage, as well as providing best practices for upgrading and health checks for vSphere environments