One of the best performing posts we’ve had this year was titled “How VMware IT Reduced Provisioning Time by 80% Using vFabric Application Director and More.” The article below is more detailed information on the project in a Q&A session format with VMware’s Director of Technology Labs and Architecture, Thirumalesh Reddy.
1. So in your previous post, you covered the business case, before and after process, project functionality and scope as well as the results. You had said provisioning time was reduced by 80% and now it's 90%. How do things keep getting better?
As we evolve and mature, we are able to automate more manual steps like load balancing VIP pool creations and post installation tasks. So, we’ve reduced provisioning time further because of this. If you think about it, most good infrastructure people already look for ways to automate redundant, manual tasks and prefer to run scripts where possible. Our technology let’s people do this on a massive scale, and we just keep finding more places to automate.
2. Besides the fact that you work for VMware and use your own products, why did the CIO or CTO really get behind this project?
Multiple times per year, we were manually provisioning or refreshing about 25 environments across our technology portfolio. We all know manual steps cause problems and are less efficient and effective. In our case, each environment provisioning or refresh cycle could take 3-8 weeks where 15-20 people touched it. Besides the extra operating expense of people’s time, a manual process with lots of hand-offs and touch points is ripe for errors and re-work. I don’t think anyone would disagree that this type of manual process is more prone to errors, incurs more delays, and includes less predictability. For our internal customers in the business and functional areas, the manual process also impacts SLAs. If you factor in compliance, risk management, and security, it’s an area ripe for improvement. Every IT shop faces these issues.
Register for VMworld!
Register for Session OPS-CIM2852 - Automated Provisioning for Business Critical Applications (Microsoft/Java) in private or public cloud:
Register for Session APP-CAP2770 - Plan, Deploy & Manage Modern Applications on vSphere Infrastructure:
Follow all vFabric updates at VMworld on Twitter:
Follow us @vmwareappmgmt:
The decision for our executives wasn’t rocket science. Since we were already operating in a 97% virtualized environment, executive management saw a reasonably sized, low risk opportunity to add capabilities and reduce costs. Even though the scope of this multi-phased project is extensive, there is no way to question the benefits we’ve already seen in the first phase.
3. In your upcoming talk at VMWorld, you say that costs were reduced by 30% in this project. Were these human resource costs, hardware, software, energy, or some other cost area? What are some examples of each cost area (human, hardware, software, energy) being reduced? How did you measure costs before and after?
Yes, you are right. I’ll explain the improvement another way. Applications have regular cycles of development. Some companies release new functionality every day, some do it twice a month, others do it quarterly or annually. In addition, there are patches and administrative releases. The point is, each cycle has a human resource cost. For some applications, it may be 10-20% of the cost. For an environment that is very complex and integrated, it can be quite a bit more. When you automate this step, it saves a significant chunk of time and operating expense. From a hardware, software, and energy perspective, we are also virtualizing the infrastructure here, and the consolidation and virtualization business case completely applies. Software costs, hardware costs, and energy costs can all go down.
To measure improvement, we looked at how many steps were removed, how much time people saved, how many fewer errors and rework – the normal process improvement types of things.
One area that should not be overlooked yet could be harder to measure is the ability to get to market faster. If you have a quarterly release cycle with a business unit (4 releases per year), and you tell them they can have 6 releases per year of the same scope, many get excited. For many companies, application improvements mean revenue increases or cost reductions – so, this stuff hits the bottom line.
4. Also in the previous post, you outlined a complex workflow with many steps and gave a high-level explanation of these steps. Could you go into some detail and give an example where there was a significant impact?
We are taking an enterprise-level approach because this is a complex ecosystem with complex apps. We listed out many apps before like Oracle ERP. It has almost 7 terabytes of DB data. In any event, we are automating each layer of each app’s stack – the web, app, database, and network, etc. This includes complete configuration and wiring. We also have the automation share bits and pieces of information across applications. A great example is probably around adding virtual VIPs – you can see the workflow in the graphic below. Here are the steps to AddVirtualVIPs:
- A Wrapper workflow provides VIP Name & Pool Member IP address
- Verify if this VIP requested exists in list of VIPs already created in F5
- If VIP exists, add the pool member to existing VIP
- If not, get next available free IP address from Men&Mice
- Add DNS entry for the new VIP in Men&Mice
- Create LTM Pool, Virtual Server & add members to the Pool
Think about the number of chats, emails, or conference calls this might take per app at your company. OK, doing it once is no big deal. Doing it for 500 production systems is considerable. Doing it for all the related environments is quite a lot of work.
5. As an example, could you tell us about how you are using vFabric Application Director with Oracle ERP and Oracle Middleware?
For Oracle ERP:
We are using vFabric Application Director to install and configure Oracle ERP 188.8.131.52. While executing any script stand-alone is not terribly amazing, having a full workflow management and monitoring system on top of all of this is cool. Here is what we can do:
- Provision VMs (Oracle Linux) - for Database, Concurrent Manager and Web/Forms
- Execute script to Restore Oracle Database and EBS Application Binaries from the Backup
- Execute script to create Database
- Execute script to Configure EBS Concurrent Manager Tier
- Execute script to Configure EBS Web/Forms Tier
- Execute script to run post-clone steps like password change.
- Execute script to Application Services
For Oracle Middleware:
We are using vFabric Application Director to install and configure Oracle SOA Suite 10gR3 and Oracle Service Bus 11g.b. Both single-node and multi-node configurations are supported. In the case of SOA, we are creating a database tier with Oracle Database 11gR2 as well. Here is our set-up:
- All Oracle installations are done using silent installation with a response file.
- SOA installation and configuration is done through cloning of an existing Oracle SOA Suite. We take a gold copy and run the apply process of the cloning to create a clone.
- OSB installation and configuration is done through fresh install of WLS and OSB followed by configuration of the endpoints and services. All configuration is done via WLST scripting.
6. From your experience working with them, what do your developers say about the change and why they like it?
They say it’s a significant transformation. Yet, there is also a huge productivity gain. For developers, much more time can be spent doing development instead of dealing with infrastructure related planning, communications, activities, and fire-drills. In some areas, self-service infrastructure is now the standard. Of course, developers don’t have to wait as much and don’t have to troubleshoot as much. Who hasn’t spent two days troubleshooting an issue to realize the error was introduced during a migration? Developers feel more productive.
7. From your experience working with them, what do your infrastructure and admin colleagues say about the change and why they like it?
People say there is a huge IT transformation from the operations standpoint. The change impacts people, process, and technology. People are more efficient, yet their focus changes. They now focus on maintaining the OS, Application Blueprints, standardization of configuration, and patch sets. Compliance and change management become immediately more effective. People spend much less time on micro-detailed, day-to-day provisioning tasks like requesting a new IP address or waiting for an install script to run. I think our team feels like they are doing less grunt work and making a bigger impact.
8. How has security, compliance, or risk management improved with this change? Could you give an example?
There is a huge improvement in security, risk, and compliance. We now have standard configurations and patch sets across the environments by using Application Director Blueprints and VCD templates. Again, change management is much more streamlined – think about what happens when you don’t need to wait 2 days for a resource to spend 2 days setting up a test environment. Or, what if you need to apply a critical patch for a widely used Linux server and could get 50 test environments for 50 different apps set up within a day?
9. How have SLAs improved? Could you give an example?
You can imagine that a truly enterprise ERP system has a wide-ranging number of stakeholders who go through the process of prioritizing what is in a release. When developers have time freed up from dealing with infrastructure, the extra time is applied to improvements. So, business units or functional groups get more functionality per release or more releases. We’ve already seen this type of improvement.
For example, we used to take an average of 3-4 weeks to provision or refresh an instance for these types of applications. Now it takes less than 24 hours. This can certainly help keep projects on-time and on-budget!
10. Anything else you’d like to mention?
No. Just to say, “Thank you for reading this!”
To learn more detail from this real-world case study, attend Thirumalesh’s presentation at VMworld in Barcelona: Cloud Application Platform Automation on vSphere Infrastructure leveraging Application Director : Real world example of running 4 billion dollar business (VMware IT)