Virtualize Business Critical Applications

We have received some requests from SAP Basis colleagues on how to go about designing SAP systems on VMware. Now that vSphere 5 can support up to 32-way virtual machines it is possible to fit  larger  SAP systems into one single virtual machine (VM) so  should we go with  2-tier versus 3-tier? Here are some guidelines.

Sizing

 First let’s cover sizing as this will impact the final VM architecture. SAP sizing is conducted in the SAP metric “SAPS” (http://www.sap.com/solutions/benchmark/measuring/index.epx ).   All SAP on VMware sizing is officially conducted by the server vendor SAP practice. VMware partners with the server vendors so we can help but we are not ultimately responsible for sizing. The background behind this is as follows:

• SAP has officially deferred sizing on physical and virtual to their hardware partners (since 1993 approx). SAP’s position is documented on SAP Marketplace https://service.sap.com/sizing   (logon credentials are required). As of April 2012:
  • Under “Sizing Responsibilities” it says “The hardware vendors are responsible for providing hardware that will meet the customer’s throughput and response time requirements”
  • Under “Virtualization - Some Statements about Sizing and Virtualization” it says “For the right virtualization strategy you should get in touch with your hardware vendor”.

• The SAPS rating per vCPU only depends on the processor model. The hardware vendor has the most up-to-date SAPS ratings of their servers so they can size most accurately. For example the SAPS rating of a virtual machine with 4 vCPUs will change if moved from one server model to another.

 The hardware vendor can conduct the sizing and provide the number of ESX servers required to fulfill the business requirements. Once this is available VMware can work with the hardware vendor and customer to jointly fine-tune the VM size and layout.

We recommend starting conservatively for business critical workloads. An initial sizing option could be to allocate number of vCPUs = number of cores on the ESX server – we would do this even for hyper-threaded systems.

To achieve higher utilizations, the total amount of vCPUs running on an ESX server can be higher than the total amount of physical cores. The ESX hypervisor is designed to optimally schedule the workload amongst the available CPUs. Additionally, it can be configured to give more important virtual machines a higher priority. Hardware supported features like hyper-threading will increase the CPU scheduling efficiency. No general statement can be made regarding the optimal CPU over-commitment ratio, as this always depends on individual utilization patterns of the workload.

 2-tier versus 3-tier

The architecture of a single SAP system consists of: a database instance; application server instances; Central Instance (CI - includes locking and message services and other SAP processes). In newer SAP releases the Central Instance is replaced with Central Services (CS - locking and messaging only) and the Primary Application Server instance (PAS). 2-tier refers to all these components running in the same guest-OS/ VM. 3-tier refers to the situation where, for a single SAP system these components are spread out onto at least two VMs.  Each of the components can be deployed into a separate VM. Advantage of 2-tier systems is that there are less VMs to manage and there is no network latency between the SAP components.

3-tier has the following advantages:

• For flexibility, better resource management and better overall high availability i.e. if everything is in one VM and the VM/guest OS / ESX server goes down you lose every component. If workload is dynamic e.g. month-end requires more app tier resource you can add/remove application server VMs as required so 3-tier is better for this (same principles as physical).

• You can set up a ESX cluster in a “n+1” setup i.e. if one ESX server goes down all the VMs can restart on remaining ESX servers and continue to perform as before (auto-restart scripts required for the instances or enter  “Autostart=1” in the instance startup profile). 3-tier setups allows you to spread the VMs for a single SAP system across multiple ESX servers so if one ESX server goes down then it minimizes the impact to a single system (off course if DB/CS virtual machine is offline the SAP system is down but hopefully only one component needs to be restarted).

• 3-tier setups allow you to size VMs better so they align with NUMA architecture.
  • DB VM – this needs to scale-up vertically so if sizing requires a large DB you can put it in a large VM. Ideally you want the VM to fit inside a NUMA node but if it can’t no big deal vSphere 5 can support a wide VM that crosses a NUMA node (and you can configure virtual NUMA to take advantage of any NUMA optimizations inside the guest OS).
  • Application server VMs can scale-out horizontally – size these in smaller blocks  such that they fit inside of a NUMA node.
  • For more background, see http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf, pages 39-40.

• ABAP+JAVA stack:   SAP has a policy whereby they prefer to separate ABAP + JAVA out - we can comply with this in virtual by putting the stacks in separate VMs.  Check out http://wiki.sdn.sap.com/wiki/display/SI/SAPs+Dual+Stack+Strategy – this recommends single stack except when it is a hard requirement in the SAP product e.g. Solution Manager.  Advantage of this is you can manage performance tuning separately in each VM, for example ABAP does not support large pages, but Java does (see SAP note 1681501 - Configure a SAP JVM to use large pages on Linux). However if you need to run a dual stack, you can in a single VM, just size the VM large enough to handle memory + CPU of both stacks.

In the physical world some customers run batch jobs on the CI which is on the same physical server as the DB instance. The advantage - the jobs run quicker as there is no network hop between the app and the DB.  In virtual a similar setup would require a large VM with the DB and app server/CI instance installed in the same guest-OS. The only downside is if there are long periods where the batch jobs are not run – we end up with an oversized VM with low utilization. Some datacenters may have a security requirement to separate the DB in its own guest-OS in which case your options are limited. VMware supports hot-add vCPU for the latest Linux and Windows versions but hot-remove is not supported.  One solution, if the batch job is designed to run in parallel threads (many SAP ABAP batch jobs have this ability), increase the degree of parallelism and distribute the batch workload across more app server VMs to decrease the overall runtime of the job (this assumes you have available CPU) – the latter can be provisioned and de-provisioned based on the cyclical nature of the workload.

Regards

Matthias Schlarb (SAP Technical Alliance Engineer)

Michael Hesse (SAP Technical Alliance Manager)

Vas Mitra (SAP Solutions Architect)

In the last few years we have had some great successes of customers virtualizing their SAP landscapes in production on VMware, check out Mazda and Miami Dade at http://www.vmware.com/solutions/partners/alliances/sap-customer-success.html, both of whom presented their case studies at SAP Sapphire and SAP Tech Ed.  For technical success follow the VMware best practice papers and SAP Marketplace notes on virtualizing on vSphere (see the references at the end of this blog).  Here are some high level guidelines and considerations.

Support

SAP has supported VMware since Dec 2007 on Netweaver 6.40 and above (64 bit) in production. VMware engineers with SAP experience work with SAP support at SAP offices to provide coordinated handling of production support tickets.

Performance

Our customers are running both OLTP and batch workloads in production. Certified OLTP SAP virtual benchmarks exist.  Two 3-tier vSphere certifications 2010016 and 2011044 (http://www.sap.com/solutions/benchmark/sd3tier.epx  ) show scalability up to 16000 and 32125 benchmark users.

If you are interested in how virtualized SAP performs in comparison to native, first consider what the chief engineer of the Starship Enterprise (NCC-1701/NCC-1701A) is fond of saying “Ye canna change the laws of physics”, neither can VMware.  For an apples-to-apples comparison take a look at 2-tier benchmark certifications 2011028 (physical) and 2011027 (vSphere5)   at http://www.sap.com/solutions/benchmark/sd2tier.epx .  The virtual result is within 6% of physical.  Note that virtualizing SAP typically occurs when there is a hardware refresh/upgrade to newer servers with much faster chips. The latter (with VMware) can provide an overall performance boost compared to the customers current environment.

SAP batch processes are intensive workloads.  Examples of such jobs include: MRP runs (Material Requirements Planning) in the manufacturing module; Demand Forecasting/planning Run in the Supply Chain Management product; Allocation Run (ARUN) in the SAP Apparel and Footwear module. SAP has designed these batch jobs to run multiple parallel units of work on multiple processes in the app servers. These app servers can be configured in multiple virtual machines that scale out horizontally to manage the overall runtime of the batch schedule (similar to horizontally scaling out OLTP workloads).  The advantage with VMware is that these app servers can be quickly deployed for month-end processing and then decommissioned once batch has completed.

Virtualize in Phases

Virtualization of SAP landscapes is not an all or nothing approach.  The SAP landscape consists of multiple SAP products (ERP, CRM, portal etc.) with separate production and non-production environments and separate application and database tiers.  Virtualize the less critical systems first, gain experience with VMware, and develop your own internal best practices and then move on to the next set of systems.  One option is to virtualize the application tier first all the way from non-production to production.

Design the Virtual Landscape Conservatively

SAP systems drive business critical processes and, as such, SAP users require and demand performance and stability.  For instance, if shipping documents cannot get processed by SAP, goods can’t leave the warehouse and the supply chain suffers. Hence SAP application owners are more likely to take a cautious approach to infrastructure changes especially if they have agreed upon performance SLAs between the IT organization and business users.  First adopt the phased virtualization approach discussed above to gain gradual confidence and experience in VMware. Next follow some conservative design practices.

You can get a sizing of your virtual environment from a VMware capacity planner assessment or from a sizing exercise by the SAP practice of the server/OEM vendor - similar to physical. VMware partners with all the server vendors. As it can be very difficult to predict actual workload before go-live, sizing will be conservative and factors in month-end peak processing requirements.  We recommend setting a memory reservation to the configured size of the virtual machine to guarantee no memory over commit. Yes this will reduce consolidation, potentially impede vMotion and requires high-density DIMMs to efficiently load an ESX server, but the goal is to minimize risk of any performance issue in production and avoid ESX swapping.  The latter can be viewed by SAP end-users as an outage and could have serious revenue impact to the business (monetary value of which far exceeds the hardware cost of the extra memory required for SAP).

Other conservative guidelines during initial architecture design may include: laying out virtual machines such that total number of vCPUs equals the number of cores of the ESX server; putting DRS in manual mode  (initially SAP admins may prefer to have manual control, also with conservative sizing DRS would not have any impact).  After go-live once the workload is known, rebalancing of virtual machines via vMotion can help increase utilizations to the point where vCPU over-commit is possible at which point automatic DRS can be activated.

 If memory of virtual machines appear to be over-sized we ask that VMware admins work with SAP Basis administrators to use SAP tools to monitor actual SAP memory usage inside the virtual machine to determine if any down-sizing is required.

Coordinated Performance Monitoring

SAP has provided a rich set of performance monitoring tools that allow Basis admins to monitor the database, OS and SAP application. All of this can be done within the SAP presentation GUI. One such SAP monitoring transaction is “OS07N” which allows monitoring of OS/guest OS level performance counters. This transaction has been updated by SAP to include some virtual level performance metrics. However certain key performance metrics in the virtualized environment such as realistic CPU usage, disk I/O and network dropped packets can only be viewed using VMware tools like vCenter performance charts. SAP admins need to be aware of these counters and should work with the VMware teams to jointly address performance monitoring and management.  The following table summarizes some useful performance metrics/tools at the different levels.

 
      Figure: Some Useful Performance Tools/Counters (WIP)

You can find the SAP on VMware best practices guide at http://www.vmware.com/files/pdf/techpaper/SAP-Solutions-on-VMware-Best-Practice-Guide-2011.pdf .  Chapter 10 has a summary of the technical best practices – yeah I know we should have separated this summary with titled sub-sections for memory, CPU, storage, network….easier to digest…..we’ll get to this on the next rev.

A complete list of VMware related SAP marketplace notes is available at http://forums.sdn.sap.com/thread.jspa?threadID=1524523&tstart=0  . As is the case with all SAP installations please read the notes.

This blog is part of a series on Virtualizing Your Business Critical Applications with VMware.  To learn more, including how VMware customers have successfully virtualized SAP, Oracle, Exchange, SQL and more, visit www.vmware.com/go/virtualizeyourapps

VMware vShield App is a hypervisor-level firewall that can be used in virtual SAP deployments to provide network isolation and facilitate quick firewall testing of SAP landscapes. SAP landscape consists of multiple SAP products (ERP, CRM, portal etc) with separate production and non-production environments.  The non-production systems can be further broken into categories such as development, sandbox, user acceptance testing, training, etc. Each landscape can consist of multiple virtual machines running different SAP databases and multiple application servers.  vShield allows these landscapes to be segregated into separate trust zones based on customer requirements.  The following diagram depicts what the security zone architecture could look like in a non production landscape example.

You can define security rules based on containers that are VMware vSphere® objects.  One such object is a vApp.  A vApp is a resource container for multiple virtual machines that work together as part of a multitier application. In the figure above, each zone can be configured as a vApp comprising all the associated virtual machines for a particular SAP landscape.  For example, a rule that denies any traffic from inside a vApp to a specific destination outside that vApp applies to all the virtual machines in that vApp.  Isolating landscapes addresses use cases like: end users with access to training systems can be protected from accessing the development and QA systems; new developers and project consultants who are coming up to speed in sandbox systems can be isolated from core development environments.

SAP administrators can benefit from vShield App in the following ways:

• Organizing SAP landscapes into vApps and setting policies against these vApps allows administrators to focus on the landscape architecture and remove themselves from details such as IP addresses.

• For SAP, an IP address change does not typically involve any change to the application configuration files—the same can be said of the firewall rules. Once firewall policies have been set against the vApp, creation of a new database and/or application server virtual machine inside of the vApp requires no extra firewall configuration. This is useful as SAP environments can be very dynamic, for example, creation of sandbox and training systems for new project personnel and end users.

•  Administrators can easily test the impact of firewall settings to the TCP/IP ports used by SAP. The SAP application uses many TCP/IP ports for communication between end-users and application servers and between application and database servers (these ports are documented at http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/4e515a43-0e01-0010-2da1-9bcc452c280b).

• Database server virtual machines can be isolated from application server virtual machines. While SAP developers mostly work within the SAP development environment, they may need access to the guest OS of the application server to work with interface files (to manage data that is transferred between separate SAP systems or between SAP and non-SAP apps). So developers may be granted access to the application server guest OS but firewall rules can block access to the guest OS of database server virtual machines which is typically restricted to the database and SAP Basis administrator.

• vShield firewall policies apply to all ESXi hosts in the cluster so they are still enforced when a virtual machine is live migrated between hosts or a to a new host in the cluster (assuming that all hosts are protected by running the vShield App appliance).

In the second part of this blog series  we will demonstrate how a vApp consisting of a 3-tier SAP landscape can be protected using vShield App.

Hello and welcome to the new VMware Business Critical Applications blog, a source for information, insight, and updates to help you virtualize your business critical applications on the VMware platform.

The blog will cover the following applications/functional areas and we will continue to add more over time:

• Microsoft Applications
  • Exchange
  • SQL Server
  • SharePoint
• SAP
• Oracle
• Java Applications 

We intend to use this blog to:

• Communicate best practices for deploying on the vSphere platform.
• Provide Updates on current solution projects and activities including solution rollouts, new collaterals etc.
• Provide Update on various product and solution enablement activities including events, webinars, partner engagements, trainings etc.
• Publish results of lab testing in the aforementioned functional areas.  Testing and results will include functional and technical use cases, workload characterization study and deployment  "how-to".  For official performance test results, please refer to the VROOM! Blog at http://blogs.vmware.com/performance/.
• Communicate general application design principles that we've discovered through research and work with our customers.
• Communicate step-by-step procedures for common application and infrastructure management tasks.
• Highlight VMware and partner products or features than can enhance the overall solution.
• Provide links to relevant online documentation.
• Provide insight on optimizing software licensing costs for virtualization.

We plan on posting regularly so grab the RSS feed or sign up for an e-mail alert to receive notification of new entries as they are posted.

For published whitepapers, including technical whitepapers and customer success stories, please visit our Business Critical Applications website at http://www.vmware.com/solutions/business-critical-apps/.  On the right-hand side, you'll find links to the individual application pages.

Thanks for reading!

Scott Salyer

Solutions Manager, VMware