In the previous post Configure DHCP and TFTP for Auto Deploy, we discussed how to setup your DHCP and TFTP servers to allow your ESX hosts to PXE boot. However, once an ESX host boots, it will need directions to know what to boot. This is where Auto Deploy Rules come in. Continue reading
Category Archives: Automation
Two new white papers are now available on the work done at Adobe on virtualizing Hadoop. The VMware-authored paper, Adobe Deploys Hadoop as a Service on VMware vSphere, focuses on the business background and justifications for virtualizing the workload. It also talks about implementing Hadoop-as-a-Service by the central Technical Operations function to satisfy the needs of the business units and data analysis groups that require Hadoop as a platform. This paper also gives details about the use of the vSphere Big Data Extensions tool which was used heavily in the project, as well as the connection to vRealize Automation that forms the basis for the cloud offering at Adobe.
The second, complementary white paper, on the same architecture, Virtualizing Hadoop in Large-Scale Infrastructures, was written by the EMC consulting team that supported the project. The EMC paper, with the title "Virtualizing Hadoop in Large-Scale Infrastructures", focuses on the technical reference architecture for the Proof-of-Concept conducted in late 2014, the results of that POC, the performance tuning work and the physical topology that was deployed using Isilon storage. The two papers were written in concert by the organizations and should be read together for a full picture of the Hadoop virtualization project. This system is now live at Adobe Digital Marketing, hosted on their Virtual Private Cloud and it is being used by different groups within the big data community there. The papers together provide an outline reference architecture for use in other installations also. Watch this space, there are more technical case studies in the works.
Speaking of technical reference material for Hadoop on vSphere, here is the current list of technical papers and websites that are now available for people to learn more about this particular subject - for your reference:
Big Data/Hadoop on VMware vSphere - Reference Materials
- Virtualizing Hadoop - a Deployment Guide
- Deploying Virtualized Cloudera CDH on vSphere using Isilon Storage - Technical Guide from EMC/Isilon or find the latest version at https://community.emc.com/docs/DOC-26892
- Deploying Virtualized Hortonworks HDP on vSphere using Isilon Storage - Technical Guide from EMC/Isilon or as above https://community.emc.com/docs/DOC-26892
- Cloudera Reference Architecture - Isilon version
- Cloudera Reference Architecture – Direct Attached Storage version
- Big Data with Cisco UCS and EMC Isilon: Building a 60 Node Hadoop Cluster (using Cloudera)
- Scaling the Deployment of Multiple Hadoop Workloads on a Virtualized Infrastructure (Intel, Dell and VMware)
Customer Case Studies
- Adobe Deploys Hadoop-as-a-Service on VMware vSphere
- Virtualizing Hadoop in Large-Scale Infrastructures – technical white paper by EMC
There are some very useful best practices in the first two technical papers.
- Virtualized Hadoop Performance with VMware vSphere® 6 on High-Performance Servers
- Virtualized Hadoop Performance with VMware vSphere 5.1
- A Benchmarking Case Study of Virtualized Hadoop Performance on vSphere 5
- Transaction Processing Council – TPCx-HS Benchmark Results (Cloudera on VMware performance, submitted by Dell)
- ESG Lab Review: VCE vBlock /systems with EMC Isilon for Enterprise Hadoop
vSphere Big Data Extensions (BDE)
- VMware BDE Documentation site
- VMware vSphere Big Data Extensions - Administrator's and User's Guide and Command Line Interface User's Guide
- Blog articles on BDE Version 2.1 - See the embedded Blogs from the Hadoop distro vendors also.
- VMware Big Data Extensions (BDE) Community Discussion
- Apache Hadoop Storage Provisioning Using VMware vSphere Big Data Extensions
- Hadoop Virtualization Extensions (HVE)
- Demos of Big Data Extensions
Other vSphere Features and Big Data
In the previous post, we covered Enabling Auto Deploy on vCenter Server Appliance 6.
There are several more steps that need to be taken to get Auto Deploy configured correctly.
In this post we discuss the next step in our journey to running Auto Deploy in your environment, which is Continue reading
I’m happy to announce that the vSphere 6 Hardening Guide Public Beta 1 is now available.
The guide is being provided as Excel spreadsheet. I’m also making a PDF doc available for easier viewing. In addition, I've also included an Excel spreadsheet of the guidelines that have moved out of the guide and into documentation. THIS IS INCOMPLETE. We are still working on some of that content. (that's why this is a beta!)
Please read the blog on the changes that have happened to the guide. LOTS of changes and the blog will explain.
Many customers are now converting over to use the vCenter Server Appliance 6.0 since vSphere 6 has reached feature parity with the Windows vCenter Server.
For those of you who are new to using the appliance, I figured I would walk you through setting up the Auto Deploy portion of the server. Continue reading
In vSphere 6.0 we now have a new concept called Exception Users. The intent of Exception Users is that they are not general admin users. I would consider them more of a “Service Account” type of access.
As a matter of fact, just the other day I got an email from someone internal at VMware that brought up a great use case for Exception Users. They were talking to a customer that wanted to access ESXi via a PowerCLI cmdlet (Get-VMHostAccount) to list out the local accounts on an ESXi server as part of their normal security reporting.
But they also wanted to enable Lockdown Mode and were finding it difficult to comply with both things. In vSphere 6.0 this is now much easier to address. Let’s get started.
User Experience ("UX") focuses on the intimate understanding of your users. What is it that they need or desire, what do they value, what are their abilities, as well as their limitations?
As you embark upon the journey to the software-defined data center (SDDC), think and architect in terms of the user experience in addition to "boxes and arrows."
- What are the desired UX outcomes for those consuming the service(s)?
- Have you considered the UX in terms of its usefulness, usability, desirability, accessibility, credibility, and its value?
In addition to fundamental tenant and business group designs, entitlements and service catalogue designs, one such area for UX consideration is the messages provided to those consuming services of the software-defined data center.
For a moment, imagine you are providing automated infrastructure delivery to multiple business segments of a large media and entertainment organization, each with their own distinct brand. The segments are built upon their individual brand and identity.
- Do you centrally brand the service that you offer or do you tailor the service to each tenant business segment?
- How would this change if instead the services were used to provide automated infrastructure delivery only to your IT Operations team and not direct end users?
The messages that appear in the inbox of the user are part of the experience. VMware vRealize Automation can send automatic notifications for several types of events, such as, the successful completion of a catalogue request or a required approval workflow. System Administrators can configure global email servers, senders and recipients that process email notifications.
Tenant Administrators can override those defaults, or add their own servers, senders and recipients if no global attributes are specified. They may even select which events will cause notifications to be sent to their users. Each component, such as the service catalog or infrastructure-as-a-service, can define events that can trigger notifications.
Additionally, each user can choose if they wish to receive notifications. Users either receive all notifications configured by the Tenant Administrator or no notifications.
Notification may also have links that allow the user to perform interactively. For example, a notification about a request that requires approval can have one link for approving the request and one for rejecting it. When a user clicks one of the links, a new email opens with content that is automatically generated. The user can send the email to complete the approval.
Messages can be easily and beautifully customized using a simple, powerful template engine. These may be customized per-locale, per-tenant, and per-notification scenario. You have the ability to define and craft the desired user experience for any notification.
Lockdown mode has been around in various forms for many releases. The behaviors have changed a few times since 5.1 with varying levels of usability success. For vSphere 6.0 we are trying to address some of these issues. Personally, what I’d love to see happen with all customers running V6.0 is that you run at a minimum the “Normal” Lockdown Mode.
I’ve been fortunate to have one of our super sharp product line managers, Alex Jauch (twitter @ajauch), spend some time explaining to me one of the new enabling technologies of vSphere 6.0: VAIO. Let's take a look at this really powerful capability and see what types of things it can enable and an overview of how it works.
VAIO stands for "vSphere APIs for IO Filtering"
This had for a time colloquially been known as “IO Filters”. Fundamentally, it is a means by which a VM can have its IO safely and securely filtered in accordance with a policy.
VAIO offers partners the ability to put their technology directly into the IO stream of a VM through a filter that intercepts data before it is committed to disk.
Why would I want to do that? What kinds of things can you do with an IO filter?
Well that’s up to our customers and our partners. VAIO is a filtering framework that will initially allow vendors to present capabilities for caching and replication to individual VMs. This will expand over time as partners come on board to write filters for the framework, so you can imagine where this can go for topics such as security, antivirus, encryption and other areas, as the framework matures. VAIO gives us the ability to do stuff to an IO stream in a safe and certified fashion, and manage the whole thing through profiles to ensure we get a view into the IO stream’s compliance with policy!
The VAIO program itself is for partners – the benefit is for consumers who want to do policy based management of their environment and pull in the value of our partner solutions directly into per-VM and indeed per-virtual disk storage management.
When partners create their solutions their data services are surfaced through the Storage Policy Based Management control plane, just like all the rest of our policy-driven storage offerings like Virtual SAN or Virtual Volumes.
Beyond that, because the data services operate at the VM virtual device level, they can also work with just about any type of storage device, again furthering the value of VSAN and VVOLs, and extending the use of these offerings through these additional data services.
How does it work?
The capabilities of a partner filter solution are registered with the VAIO framework, and are surfaced for user interaction in the SPBM Continue reading
If you already heard the exciting news about VMware new offerings - vSphere 6, Virtual SAN 6 and vSphere Virtual Volumes - and thought it can't get any better, we have a small surprise for you. Virtual SAN 6.0 includes a host of new features including high performance snapshots and clones, all flash Virtual SAN with intelligent 2-tier model, failure domains and more. One of the most requested features among Virtual SAN 5.5 customers is enabling greater visibility to what happens "under the hood". The Virtual SAN team developed a new health dashboard that will help customers tackle underlying hardware issues as part of the v6.0 release, but in addition to that the vRealize operations team developed an advanced set of dashboards aimed at making Virtual SAN users life much easier.
The Virtual SAN team along with the vRealize Operations team are thrilled to offer you a unique opportunity to beta test the new vRealize Operations Management Pack for Storage Devices. The management pack will feature advanced insight into Virtual SAN through advanced analytics to enable rapid troubleshooting and cluster optimization.
We will share more details closer to the start of the beta program which will kick off in Q1 2015. If you would like to get more information and be on the list of people who get an invitation to participate in the beta, please sign up here – www.vmware.com/go/vrops4vsan-beta
Check out the vROps MPSD blog post for more details - http://blogs.vmware.com/management/2015/02/vsan-simplifying-sddc-storage-operations-with-vrealize-operations-management-pack-for-storage-devices.html