In the previous post Configure DHCP and TFTP for Auto Deploy, we discussed how to setup your DHCP and TFTP servers to allow your ESX hosts to PXE boot. However, once an ESX host boots, it will need directions to know what to boot. This is where Auto Deploy Rules come in. Continue reading
Category Archives: vCenter Server
vCenter Server has become a mission critical part of most virtual infrastructures. It can be a single point of failure if it is not designed for availability. vCenter Server 6 has many changes relating to vCenter Server and its components and careful consideration has to be made in the design of its architecture.
There are multiple solutions for high availability. Many of these options can be combined to provide different levels of availability. vSphere HA, FT, vCenter Watchdog services and in guest clustering solutions can be combined depending on customer requirements for availability.
The Platform Services Controller (PSC) serves many VMware solutions in addition to vCenter Server such as VROPS, View, etc. The PSC deployment modes have to be carefully evaluated based on unique customer requirements and architected appropriately as well.
The VMware vCenter Server 6.0 Availability Guide is a great resource for architecting a HA solution for vCenter Server. I hope you find it useful!
Two new white papers are now available on the work done at Adobe on virtualizing Hadoop. The VMware-authored paper, Adobe Deploys Hadoop as a Service on VMware vSphere, focuses on the business background and justifications for virtualizing the workload. It also talks about implementing Hadoop-as-a-Service by the central Technical Operations function to satisfy the needs of the business units and data analysis groups that require Hadoop as a platform. This paper also gives details about the use of the vSphere Big Data Extensions tool which was used heavily in the project, as well as the connection to vRealize Automation that forms the basis for the cloud offering at Adobe.
The second, complementary white paper, on the same architecture, Virtualizing Hadoop in Large-Scale Infrastructures, was written by the EMC consulting team that supported the project. The EMC paper, with the title "Virtualizing Hadoop in Large-Scale Infrastructures", focuses on the technical reference architecture for the Proof-of-Concept conducted in late 2014, the results of that POC, the performance tuning work and the physical topology that was deployed using Isilon storage. The two papers were written in concert by the organizations and should be read together for a full picture of the Hadoop virtualization project. This system is now live at Adobe Digital Marketing, hosted on their Virtual Private Cloud and it is being used by different groups within the big data community there. The papers together provide an outline reference architecture for use in other installations also. Watch this space, there are more technical case studies in the works.
Speaking of technical reference material for Hadoop on vSphere, here is the current list of technical papers and websites that are now available for people to learn more about this particular subject - for your reference:
Big Data/Hadoop on VMware vSphere - Reference Materials
- Virtualizing Hadoop - a Deployment Guide
- Deploying Virtualized Cloudera CDH on vSphere using Isilon Storage - Technical Guide from EMC/Isilon or find the latest version at https://community.emc.com/docs/DOC-26892
- Deploying Virtualized Hortonworks HDP on vSphere using Isilon Storage - Technical Guide from EMC/Isilon or as above https://community.emc.com/docs/DOC-26892
- Cloudera Reference Architecture - Isilon version
- Cloudera Reference Architecture – Direct Attached Storage version
- Big Data with Cisco UCS and EMC Isilon: Building a 60 Node Hadoop Cluster (using Cloudera)
- Scaling the Deployment of Multiple Hadoop Workloads on a Virtualized Infrastructure (Intel, Dell and VMware)
Customer Case Studies
- Adobe Deploys Hadoop-as-a-Service on VMware vSphere
- Virtualizing Hadoop in Large-Scale Infrastructures – technical white paper by EMC
There are some very useful best practices in the first two technical papers.
- Virtualized Hadoop Performance with VMware vSphere® 6 on High-Performance Servers
- Virtualized Hadoop Performance with VMware vSphere 5.1
- A Benchmarking Case Study of Virtualized Hadoop Performance on vSphere 5
- Transaction Processing Council – TPCx-HS Benchmark Results (Cloudera on VMware performance, submitted by Dell)
- ESG Lab Review: VCE vBlock /systems with EMC Isilon for Enterprise Hadoop
vSphere Big Data Extensions (BDE)
- VMware BDE Documentation site
- VMware vSphere Big Data Extensions - Administrator's and User's Guide and Command Line Interface User's Guide
- Blog articles on BDE Version 2.1 - See the embedded Blogs from the Hadoop distro vendors also.
- VMware Big Data Extensions (BDE) Community Discussion
- Apache Hadoop Storage Provisioning Using VMware vSphere Big Data Extensions
- Hadoop Virtualization Extensions (HVE)
- Demos of Big Data Extensions
Other vSphere Features and Big Data
In the previous post, we covered Enabling Auto Deploy on vCenter Server Appliance 6.
There are several more steps that need to be taken to get Auto Deploy configured correctly.
In this post we discuss the next step in our journey to running Auto Deploy in your environment, which is Continue reading
I have a confession.
My data center kit has been using too much energy.
Having kit available at my disposable is great, but I have been wasting this resource when it's not required by my workloads. And if there's one thing I try to be conscious of, it's energy consumption. Just ask my kids who I chase from room to room turning off lights, screens, and the lot when they aren't using them.
But why not in the data center? Did you know that hosts typically use 60%+ of their peak power when idle?
Until recently, I had overlooked configuring my kit to use the vSphere Distributed Power Management ("DPM") feature to manage power consumption and save energy.
With the release of vSphere 6.0 it's a good time to review and take deeper look into the capabilities and benefits of this feature.
What is VMware vSphere Distributed Power Management?
VMware vSphere Distributed Power Management is a feature included with vSphere Enterprise and Enterprise Plus editions that dynamically optimizes cluster power consumption based on workload demands. When host CPU and memory resources are lightly used, DPM recommends the evacuation of workloads and powers-off of ESXi hosts. When CPU or memory resource utilization increases for workloads or additional host resources are required, DPM powers on a required set of hosts back online to meet the demand of HA or other workload-specific contraints by executing vSphere Distributed Resource Scheduler ("DRS") in a "what-if" mode. DRS will ensure host power recommendations are consistent with the cluster constraints and resources being managed by the cluster.
Beneath the covers there are key challenges that DPM addresses to enable effective power-savings capabilities:
- Accurately Assessing Workload Resource Demand
- Avoiding Frequent Power-on/Power-off of Host and Excessive vMotion Operations
- Rapid Response to Workload Demand and Performance Requirements
- Appropriate Host Selection for Power-on/Power-Off within Tolerable Host Utilization Ratios
- Intelligent Redistribution of Workloads After Host Power-on/Power-Off
Once DPM determines the number of hosts needed to satisfy all workloads and relevant constraints, and DRS has distributed virtual machines across hosts to maintain resource allocation constraints and objectives, each powered-on host is free to handle its power management
Hosts Entering and Exiting Standby
When a host is powered-off by DPM, they are marked in vCenter Server in "standby" mode indicating that they are powered-off but available to be powered-on when required. The host icon is updated with a crescent moon overlay symbolizing a "sleeping" state for the host.
DPM can awaken hosts from the standby mode using one of three power management options:
- Intelligent Platform Management Interface (IPMI)
- Hewlett Packard Integrated Lights-Out (iLO), or
- Wake-On-LAN (WOL).
Each protocol requires its own hardware support and configuration. If a host does not support any of these protocols it cannot be put into standby by DPM. If a host supports multiple protocols, they are used in the following order: IPMI, iLO, WOL. This article is focused on the use of the first two.
Many customers are now converting over to use the vCenter Server Appliance 6.0 since vSphere 6 has reached feature parity with the Windows vCenter Server.
For those of you who are new to using the appliance, I figured I would walk you through setting up the Auto Deploy portion of the server. Continue reading
With the announcement of vSphere 6 becoming Generally Available, I figure it a good time to shine some light on some of the updated features of Host Profiles. Host Profiles allow you to establish standard configurations for your ESXi hosts and to automate compliance to these configurations, simplifying operational management of large-scale environments and reducing errors caused by misconfigurations. In this release we've made several improvements which will make updates and applying of Host Profiles easier and with minimal disruption.
What's New Continue reading
I often get questions around backing up and restoring vCenter Server with vSphere Data Protection (VDP). VDP includes an "emergency restore" feature, which enables the restoration of a virtual machine even if vCenter Server is offline. This makes VDP a possible option for backing up a vCenter Server virtual machine and restoring it if it is lost, e.g., deleted from the datastore on which it resides. In my environment, I have a 4-node vSphere 6.0 cluster running approximately 10 virtual machines including the vCenter Server virtual machine (Windows OS) on a VMware Virtual SAN datastore. I just powered off and deleted the vCenter Server virtual machine. Let's see what happens...
Architectural changes to vSphere 6:
vCenter Server 6 has some fundamental architectural changes compared to vCenter Server Server 5.5. The multitude of components that existed in vCenter Server 5.x has been consolidated in vCenter Server 6 to have only two components vCenter Management Server and Platform Services Controller, formerly vCenter Server Single Sign-On.
The Platform Services Controller (PSC) provides a set of common infrastructure services encompassing
- Single Sign-On (SSO)
- Certificate Authority
The vCenter Management Server consolidates all the other components such as Inventory Service & Web Client services along with its traditional management components. The vCenter Server components can be typically deployed in with either embedded or external PSC. Care should be taken to understand the critical differences between the two deployment models. Once deployed one cannot move from one mode to another in this version.
Over the course of the last few months I've been working on a pretty massive deployment guide for vCenter Server 6, the result turned into a 100 page guide. Before getting scared off by the size the guide it goes into details for installing and upgrading many different scenarios including new installs and upgrades from the most common configurations.