Home > Blogs > VMware vSphere Blog > Category Archives: Uncategorized

Category Archives: Uncategorized

Crash consistent backups and database cloning with Virtual Volumes

In the first part of this series we provided a high level view of the benefits of using Virtual Volumes enabled storage for database operations. In the second part of this series we examined in more detail how Virtual Volumes can improve the backup and recovery capabilities for business critical databases, specifically Oracle.

The backups for Oracle can be Database consistent or Crash consistent. In this part we will look at Crash consistent backup and recovery and also how database cloning is simplified by the use of VVol. Continue reading

Architecting Virtual SAP HANA Using VMware Virtual Volumes And Hitachi Storage

VMWorld Recap: SAP HANA and VMware Virtual Volumes

This is a follow up to my earlier VMWorld blog; “Virtualizing SAP HANA Databases Greater Than-1TB On vSphere-5-5”, where I discussed SAP Multi-Temperature Data Management strategies and techniques which can significantly reduce the size and cost associated with SAP HANA’s in-memory footprint. This blog will focus on Software-Defined Storage and the need for VMware Virtual volumes when deploying Mission Critical Applications/Databases like SAP HANA as discussed in my VMWorld session.

Multi-Temperature Data Management Is By Definition Software-Defined Storage

SAP and VMware customers who plan on leveraging multi-temperature strategies, where data is classified by frequency of access as either hot, warm or cold depending on data usage is the essence of Software-Defined Storage. This can also be equated to EMC’s Information Lifecycle Management which examines the value of data to the business over time. To bring the concept of the Software-Defined Data Center and more precisely Software-Defined Storage to reality, see Table 1. This table depicts the various storage options for SAP HANA so customers can create an architecture that aligns with the business and its applications demands.

Table 1: Multi-Temperature Storage Options with SAP HANA


Planning Your Journey To Software-Defined Storage

As we get into the various storage options for SAP HANA, VMware has made it very easy to create and deploy software defined storage in the form of Virtual Volumes. However I want to stress the actual definitions of how the storage should be abstracted is a collaborative task, at a minimum you must involve the storage team, VI-Admins, application owners, and dba’s in order to create an optimized virtual architecture; this should not be a siloed task.

In my previous post I discussed the storage requirements for SAP HANA In-Memory, Dynamic Tiering, Near-Line Storage, and the Archiving Components; one last option I did not cover in Table 1 is Data Aging which is specific to SAP Business Suite. Under normal operations SAP HANA does not preload data into memory, data is loaded upon first access, so the first time you access data its always off disk.

With Data Aging you can essentially mark data so its never loaded into memory and will always reside on disk. This is not available on all modules for Business Suite, so please check with SAP for availability and roadmap with respect to Data Aging.

Essentially this is another SAP HANA feature which enables customers to reduce and manage their memory footprint more efficiently and effectively. The use of Data Aging can change the design requirements of your Software-Defined Storage, if Data Aging becomes more prevalent in your SAP Landscape, VMware Virtual Volumes can be used to address the changing storage requirements of the application by seamlessly migrating data between different classes of software-defined storage or VMDKs.

VMware Virtual Volumes Transform Storage By Aligning With SAP HANA’s Requirements

Now lets get into Virtual Volumes and the problems they solve, with Virtual Volumes the fundamental model is centered around provisioning storage based on the application needs rather than the underlying infrastructure. When deploying SAP HANA using the Tailored Data Center Integration model, the storage KPIs can be quite complex, so how do customers translate latency, throughput for reads – writes – and updates, at various block sizes to the storage layer?

Plus how does a customer address the storage requirements for SAP HANA’s entire data life cycle, whether you are planning on using Dynamic Tiering, with or without Near-Line-Storage and what is the archiving strategy storage requirements as well. Also some of the storage requirements do tie back to the compute layer, as an example with Dynamic Tiering if you plan on using Row Level Versioning there is a compute to memory relationship for storage that comes into play when sizing

Addressing and achieving these design goals using an infrastructure centric model can be quite difficult because you are tied to physical LUNs and trust me, with mission critical databases, you will always have database administrators fighting over LUNs with the lowest numbers because of the concerns around radial density. This leads to tremendous waste when provisioning storage using an infrastructure centric model.

VMware Virtual Volumes significantly reduces the storage design complexity by using an Application Centric model because you are not dealing with storage at the LUN level, instead vSphere admins use policies to express the application requirements to the storage array, then the storage array maps storage containers to the application requirements.

What are VMware Virtual Volumes?

At a high level I’ll go over the architecture and components of Virtual Volumes, this blog is not intended to be a deep dive into Virtual Volumes, instead my goal is to convey that mission critical uses cases for VVOLS and software-defined storage are real. For an excellent white paper on Virtual Volumes see; “VMware vSphere Virtual Volumes Getting Started Guide”.

As shown in Figure 1., Virtual Volumes are a new type of virtual machine object which are created and stored natively on the storage array. The Vendor Provider also known as the VASA Provider, which are the vSphere Storage APIs for Storage Awareness (VASA) that provide the storage awareness services and mediates out of the box communications between vCenterServer and EXi Hosts on one side and the storage system on the other side.

The storage containers are pools of raw storage that a storage system can provide to virtual volumes and unlike LUNS and NFS, they do not require pre-configured volumes on the storage side. Also with virtual volumes you still have the functionality you would expect when using native VMDKs

Virtual Datastores represents a storage container in a vCenter Server instance, so it’s a 1:1 mapping to the storage systems storage container. The ESXi Hosts have no direct access to the virtual volumes on the storage side, so they use a logical I/O proxy called a protocol endpoint and as you would expect VVOLs are compatible with industry standard protocols, iSCSI, NFS, FC, and FCoE

The Published Storage Capabilities will vary by storage vendor depending on which capabilities have been exposed and implemented. In this blog we will be looking at the exposed capabilities of Hitachi Data Systems like latency, throughput, Raid Level, Drive Type/Speed, IOPS, and Snapshot frequency to mention a few.

Figure 1: vSphere Virtual Volumes Architecture and Components


VMware HDS: Creating Storage Containers, Virtual Volumes, and Profiles for Virtual SAP HANA

Now Virtual Volumes are an Industry-wide Initiative, essentially a who’s who of the storage industry are participating in this initiative, however this next section will be representative of the work done with Hitachi Data Systems

And again the guidance here is collaboration when architecting software-defined storage for SAP HANA landscapes and for that matter any mission critical application or database. Because the beauty of software defined storage is once created and architecture correctly you can then provision your virtual machines in an automated and consistent manner.

So in the spirit of collaboration, I got together with Hitachi’s SAP alliance team, their storage team, and database architects and we came up with these profiles, policies, and containers to use when deploying SAP HANA landscapes.

We had several goals when designing this architecture; one was to use virtual volumes to address the entire data life cycle of SAP HANA, the in-memory component, Dynamic Tiering, Near-Line storage, and archiving or any supported combination of the above when creating a SAP HANA landscape. And secondly we wanted to enable rapidly provisioning of SAP HANA landscapes, so we created profiles, policies, and containers which could be used to deploy SAP HANA databases whose in-memory component could range from 512GB to 1TB in size.

I’ll review some of the capabilities HDS exposed which were used for this architecture:

  • Interestingly enough we were able to meet the SAP HANA in-memory KPIs using Hitachi Tier 2 storage which consisted of 10K SAS drives for both log and data files, as well as for the Operating System and the SAP HANA shared file system. This also simplified the design. We then used high density SAS drives for the backup areas
  • We enabled automatic storage managed snapshots for HANA data, log and the OS; and set the Snapshot frequency based on the classifications of Critical, Important, or Best Effort.
  • So snapshots for the data and log were classified as Critical while the OS was classified as Important and the backup area we didn’t snapshot at all
  • We also tagged this storage as certified, capturing the model and serial number, since the SAP HANA in-memory component requires certified storage. We wanted to make sure that when creating HANA VM’s you’re always pulling from certified storage containers.
  • The Dynamic Tiering and NLS storage had similar requirements so could be provisioned from the same containers and since these are disk based columnar databases we selected Tier 1 storage SSDs for the data files based on the random read/write patterns
  • And stuck with SAS drives for the log files since sequential workload don’t benefit much from SSDs. Again because of the disk based access we selected Tier 2 to satisfy the IOPS and Latency requirements.
  • Then finally for the archiving containers we used the lowest cost & highest density storage, pretty much just a file system.

Now there’s just too much information to cover in this effort with HDS but for those of you interested, VMware and Hitachi we will be publishing a Co-Logo White Paper which will be a much deeper dive into how we architected these landscapes so customers can do this almost out of the box.

Deploying VMware Software-Defined Storage With vSphere and Hitachi Command Suite

Example: SAP HANA Dynamic Tiering and Near-Line Storage Tiers. These next couple of screen captures will show how simple virtual volumes are to deploy once architected correctly

Figure 2: Storage Container Creation: SAP HANA DT and NLS Tier


Figure 3: Create Virtual Machine Storage Policies SAP HANA DT/NLS Data/Log File


Figure 4: Create New SAP HANA DT VM Using VVOLS Policies With Hitachi Storage


Addressing Mission Critical Use Cases with VMware Software-Defined Storage

SAP HANA and Multi-Temperature Data Management is the poster child for mission critical software-defined storage use cases. VMware Virtual Volumes solves the complexities and simplifies storage provisioning by using an application centric model rather than an infrastructure centric model.

The SAP HANA in-memory component is not yet certified for production use on vSphere 6.0, however Virtual Volumes can be used for SAP HANA Dynamic Teiring, Near-Line Storage, and Archiving. So my advice to our customers is to start architecting now, get together with your storage admins, VI Admins, application owners, and database administrators to create containers, policies, and profiles correctly so when vSphere 6.0 is certified you are ready to “Run SAP HANA Simple”.



Database-as-a-Service (DBaaS): Reference Architecture with VMware and Tintri

Database-as-a-Service (DBaaS) is a real-world example of cloud computing that delivers databases and database applications through self-service portals without IT intervention. Providing multiple copies of relational database servers for testing and development is traditionally a complex operation that involves the combined efforts of multiple teams and the creation of custom scripts. Yet the ability to quickly provision instances of Oracle and SQL Server databases can reduce the time to create, test, deliver, and deploy new applications.

A study of DBAAS with VMware vRealize Automation (vRA), Tintri storage done by VLSS highlights the capabilities of these platforms to do this efficiently and in an automated manner. The DBaaS reference architecture is only useful if end-users can deploy a catalog of VMs under varying conditions and workloads. These test results highlight three accomplishments in the vRA DBaaS reference architecture with Tintri.

1. VAAI enabled offload allowed the Tintri storage array cloning feature to drop the average time to provision 50 VMs from 17 minutes to 6 minutes; a 65% drop.
2. The DBaaS provisioning times were low and stable even as the load is increased significantly on the DBaaS systems.
3. Even when an extreme workload is applied to the Tintri VMstore storage array, the DBaaS provisioning times were consistently low.

The attached White Paper goes into detail on this study.

Virtual Volumes: A game changer for operations of virtualized business critical databases

This is first of a series of posts on deploying vSphere Virtual Volumes for Tier 1 Business Critical Databases. Although this article is written with a focus on Oracle databases, much of this discussion holds good for any Mission critical application.
Business critical databases are among the last workloads virtualized in enterprises, primarily because of the challenges that they pose with growth and scale. Typically the low hanging fruits are virtualizing the Development, Testing/QA, Staging databases after running a successful POC and then moving on the big guy’s i.e. the Production databases.

There are many common concerns about virtualizing business critical databases that inhibit and delay virtualization of these workloads:
• Business critical virtualized databases need to meet strict SLAs for performance and storage has traditionally been the slowest component
• Databases grow quickly, while at the same time there is a need to reduce backup windows and their impact on system performance.
• There is a regular need to clone and refresh databases from production to QA and other environments. However, the size of the modern databases make it harder to clone and refresh data from production to other environments
• Databases of different levels of criticality need different storage performance characteristics and capabilities.
• There is a never-ending debate between DBAs and Systems administrators regarding filesystems VS raw devices and VMFS VS RDM. These are primarily due to some of the deficiencies that existed in the past with virtualization.
Levels of database operations on VMware environments

Generally speaking there are 3 levels from which regular database operations (i.e. backup, cloning, etc.) can be triggered: application level, vSphere level, and storage level.

Furthermore, each approach has benefits but also drawbacks. For instance, application level operations (Oracle RMAN, SQL) may provide finer operation granularity but performance is not optimal. vSphere level operations offer VM granularity but a VM level snapshot will stun a VM for some time during snapshot coalescing/deletion (KB 1002836: A snapshot removal can stop a virtual machine for long time). Finally, storage level operations offer better performance but lack VM granularity as operations are executed at LUN level.

The ideal solution to address database operation challenges

An ideal solution would combine the built-in storage capabilities with the granularity of VM-level operations, like snapshots. More specifically:
• The solution should be able to trigger backups and clones with VMDK granularity at the same time.
• Do a storage level snapshot triggering the operation at the VM level, which is the fastest and the ideal among all the three above solutions.
• The solution would allow different database components to be aligned with different storage data services needed.

Technology Preview: Enriching vSphere with hybrid capabilities


Today VMware is revealing a Technology Preview of Project SkyScraper, a new set of hybrid cloud capabilities for VMware vSphere that will enable customers to confidently extend their data center to the public cloud and vice-a-versa by seamlessly operating across boundaries while providing enterprise-level security and business continuity.

At VMworld, we will demonstrate live workload migration with Cross-Cloud vMotion and Content Sync between on-premises and vCloud Air.  These features will complement VMware vCloud® Air™ Hybrid Cloud Manager™ – a free, downloadable solution for vSphere Web Client users, with optional fee-based capabilities. Hybrid Cloud Manager consolidates various capabilities such as workload migration, network extension and improved hybrid management features into one easy-to-use solution for managing workloads in vCloud Air from the vSphere Web Client.

Cross-Cloud vMotion is a new technology based on vSphere vMotion that allows customers to seamlessly migrate running virtual machines between their on-premises environments and vCloud Air. Cross-cloud vMotion can be used via the vSphere Web Client, enabling rapid adoption with minimal training. The flexibility provided by this technology gives customers the ability to securely migrate virtual machines bi-directionally without compromising machine up-time; all vMotion guarantees are maintained.

Content Sync will allow customers to subscribe to an on-premise Content Library and seamlessly synchronize VM templates, vApps, ISOs, and scripts with their content catalog in vCloud Air with a single click of a button. This feature will ensure consistency of content between on-premise and the cloud, eliminating error prone manual sync process.

Learn more about these two capabilities under Project Skyscraper by visiting us the VMware booth at VMworld 2015.

Oracle on vSphere book – Tech Target Interview of Authors

Tech Target has completed and published an interview of the authors (Don Sullivan and Kannan Mani) of the Oracle on vSphere VMware press book.  The published interview is linked  below:

The official VMware press book and the definitive authority on the subject of Oracle on vSphere: http://www.amazon.com/Virtualizing-Oracle-Databases-vSphere-Technology/dp/0133570185 “Serious Databases Require Serious Virtualization”

— Putting Oracle databases on a virtualized infrastructure – http://searchvmware.techtarget.com/feature/Putting-Oracle-databases-on-a-virtualized-infrastructure — The perks to virtualizing Oracle on vSphere 6 – http://searchvmware.techtarget.com/feature/The-perks-to-virtualizing-Oracle-on-vSphere-6

Learning about OpenStack and VMware Integrated OpenStack


If you are a vSphere Administrator looking to learn about OpenStack, we’ve got you covered.

The OpenStack team at VMware has created multiple resources to help your overall knowledge of the OpenStack framework, and specific working principles and capabilities of our VMware Integrated OpenStack distribution. These resources include:

1-    Online VMware Integrated OpenStack Fundamentals training – this free online class includes an introduction to OpenStack (2~3 Hours) [registration required]

2-    VMware Integrated OpenStack YouTube Playlist check out our growing list of video tutorials

3-     VMware Integrated OpenStack product walkthrough – Product walkthroughs provide a guided step-by-step screenshots for installing, configuring and managing VMware Integrated OpenStack

4-    VMware Integrated OpenStack Hands-on-Lab – This Hands-On-Labs will give you the opportunity to have a hands on experience with VIO on a VMware-hosted Lab infrastructure.

5-    VMware Integrated OpenStack documentation center – Official Documentation center that includes Quick Start Guide, User Guide and Administration Guide

6-    Watch VMware related Sessions at the OpenStack summit in Vancouver (May 2015)

7-    Check our blog or Follow our twitter handle for the latest news and resources @vmware_os

8-    Reach out to VMware Professional Services and hire an OpenStack gunslinger to sit side-by-side with the team for immersive mentoring; available in 2 week, 4 week and 8 week sessions.

Please check out these great resources, and you will understand why we belive that the best way to run OpenStack is on top of vSphere and NSX

Also, we’d love feedback on these resources and any additional content that would be most valuable to a vSphere admin looking to learn more about OpenStack. And don’t forget, VMware Integrated OpenStack is free if you have vSphere Enterprise Plus, vSphere with Operations Management Enterprise Plus and vCloud Suite. You can download it from here  [Myvmware Login required].

Happy learning,

Amr Abdelrazik

Agile Big Data Webcast

We are adding a special webcast to the vSphere 6 Webcast series.

The next webcast will be on the topic of agile Big Data happening on June 23 at 9am PST.

Learn how vSphere 6 + Big Data Extensions 2.2 can help you build an agile Big Data platform.      Provision your Hadoop clusters faster, simplify configuration and management of your nodes, efficiently scale and utilize resource saving you time and money, all with no impact on performance.  Justin Murray, our Big Data guru, will be sharing his insights and answer any questions you may have.

Register for the free live event today!

vSphere 6 Security Webinar

Just a reminder on the vSphere 6 Webcast series.

The next webcast will be on the topic of vSphere 6 Security happening on June 16 at 9am PST.

Security at the top of mind? Learn about security in vSphere 6 including considerations for Platform Services Controller. Mike Foley our security guru will be sharing his insights and showing off what’s new.

Register for the event today!

Big Data Extensions Version 2.2 – What’s New? A summary of the new features.

The new  vSphere Big Data Extensions Version 2.2 shipped on the 5th June 2015!

Here is a quick summary of the new features that appear in the 2.2 release. This is an exciting and much-awaited release. As always, refer to the technical documents and the release notes to get more detail on these subjects. 

• Support for the Latest Hadoop Distributions. BDE 2.2 supports the latest versions from the major Hadoop distribution vendors, including Bigtop 0.8, Cloudera CDH 5.4, Hortonworks HDP 2.2, MapR 4.1, and Pivotal PHD 3.0.

• Better Fully Qualified Domain Name (FQDN) Management. We found that some users had difficulty with generating FQDNs within their network for newly cloned virtual machines. BDE can now generate and propagate meaningful host names in FQDN form for your new  virtual machines that host the Hadoop nodes. The new FQDNs will be registered to a DNS server if you are using a Dynamic DNS server.

• Shrink clusters. You can now reduce (as well as expand) the number of worker virtual machines that belong to a running Hadoop cluster in an easy way. The virtual machines targeted for shrinking will be quiesced, withdrawn from the Hadoop cluster and then deleted to release any resources that they used completely.

• Active Directory/Lightweight Directory Access Protocol (AD/LDAP) integration. You can use an AD/LDAP server to manage the accounts generated by BDE within the Hadoop nodes . You can specify the accounts to be Hadoop users accounts and/or service accounts in an AD/LDAP server.

• vSphere 6.0 Instant Clone. BDE will, at the user’s request, use Instant Clone technology to spin up new Hadoop VMs. This feature reduces the time of spinning up Hadoop VMs and the runtime footprint. This is an optional way to do this. You can choose to use the older “full clone” method also if you prefer to. We recommend that you use this new type of cloning for your test and development workloads to begin with.
• Centralised logging. You can configure BDE to direct logging information to an external syslog server including LogInsight.
• Quiesce the BDE management server. You can quiesce BDE management server with a command so that you can backup BDE management server’s data for your clusters safely.
• Automatic GUI installation. BDE GUI is automatically registered to the vCenter after BDE is deployed.

• Support for the Latest Partner Hadoop Management Tools. BDE 2.2 supports Cloudera Manager 5.3, and Ambari 1.7. You have more flexibility to deploy Hadoop clusters, including a compute-only cluster,a  HBase-only cluster,a  data-compute separated cluster etc. even when using a Partner Hadoop Management tool.

• Support for the Latest Isilon Version. Fully automated process to deploy and manage compute only clusters on OneFS 7.2.

• Big Data Extensions Upgrade. You can upgrade Big Data Extensions 2.1 to the current version, Big Data Extensions 2.2, and preserve all the data for the Hadoop clusters that were created using Big Data Extensions 2.1. All of your existing clusters can be managed by Big Data Extensions once the upgrade completes.

• Localization. BDE is localized to 6 languages including DE, FR, ZH_CN, ZH_TW, KO, and JA.