Last month at VMworld US, I had an opportunity to deliver a session with Trent TeSelle (@tteselle), DevOps Engineer and a technical lead at Fannie Mae. The topic of the session was IT Automation with vRealize Automation, and more specifically the journey to implementing an IT automation platform and the benefits that come with that.
Fannie Mae is a leading source of financing for mortgage lenders, providing access to affordable mortgage financing in all markets. Their financing makes sustainable homeownership and workforce rental housing a reality for millions of Americans.
Trent is the technical lead for their Software-Defined Datacenter (SDDC) solution built on VMware vRealize Automation and Puppet Enterprise. He has over 20 years of Enterprise IT experience in various Systems Integration, Architecture, and DevOps Engineering roles, designing and implementing IT solutions for business success. When you hear him speak you see how excited he is about this topic!
RapidIT, the internal branding for automated delivery and management of IT infrastructure, has been developed and managed by the Hosting Services engineering team. RapidIT utilizes vRealize Automation, Puppet Enterprise and ServiceNow to provide self-service provisioning and management of all virtual machines in their private cloud infrastructure.
Getting Started
They started the project in the end of 2014 with an agile approach – deliver value quickly then move on. At each stage, they were looking for the “Best Bang for the Buck”, the next item or process to be streamlined with automation and process improvement. Their guiding principle was 80/20 – automate 80% of the job. Do not automate exceptions. Before they started, it took them roughly 60 days and 33 hand-offs to delivered a virtualized environment to their end users.
Image 1: Fannie Mae, before RapidIT: 60 days and 33 hand-offs to deliver virtual infrastructure environments
The first step was to automate the creation of VMs. This cut off 5 days and removed one hand off – a good start! Next, they automated the creation of the application custom filesystem, creation of database (this was a legacy process they had to work out with the database team) and the app/server tier. They are now down to 35 days to deliver an environment, with “only” 19 hand-offs. What a progress! Along the way, they also streamlined existing processes by eliminating the requirement to open change tickets, since that was not needed anymore.
Image 2: Automating 4 steps and eliminating one that was no longer needed through automation
When Process is a Blocker
Trent and his team continued to automate and streamline the delivery stack. Until they were not able to anymore. Existing legacy processes became a blocker. They realized that automation by itself can take you only so far. Process changes were required to achieve success. With management support, they started an IT reorg initiative to streamline IT processes and deliver fully automated self-service.
Excel spreadsheet was replaced by integrating ServiceNow directly with vRealize Automation. This allowed to automatically generate the change tickets that were needed when a service was requested through the ServiceNow catalog. They addressed most of the offerings and requests – remember: 80/20 rule. Do not automate exceptions. Creating a standard catalog was one of the biggest wins in terms of reducing the number of days it takes to provision a service, and the number of hand-offs.
Image 3: Almost there. Down to 3 days to deliver an environment with 2 hand-offs
At this point, they reached a (almost) fully automated self-service delivery. The accounting approval process was now holding them up, which delayed the delivery of new environments. Eventually, pre-approval process was replaced with (1) policy-based automated approval for spending amounts under certain limits, and (2) post spending notifications.
Image 4: Down to same day delivery of new environments
Results
That’s it! They are now down from 12-16 weeks to same day delivery of new environments. The economical savings over two years are amazing: 3,800 VMs X $898* per VM = total $3.4M.
One of the things they have learned along the way was that implementing automation itself can be the easy part. Changing or replacing legacy processes was a challenge, especially when coordination and collaboration with other teams was needed. However, that was key to their success. Often times they handled this with metrics. Metrics not only show improvements, but they are also crucial in getting management support for the change.
* Based on Forrester TEI Study for VMware vRealize Automation
Try our vRealize Automation Hands-On Lab here!
