This blog was co-authored by Glen Tindal, vice president of the solutions business unit at Capstone IT, and Pre Bhakta, staff solution engineer at VMware.
One may say that competitive dynamics in the automotive space are constantly changing, and they would not be wrong in the slightest.
As retailers face the daily battlefield of getting their customers’ attention with the right mix of products at the right price and on the right platform, their technology infrastructure must be lean and efficient while achieving those goals—and that is no easy feat.
One of our customers, an automotive company in the U.S., needed to speed up time to revenue in the face of competition and their own increasing IT infrastructure costs.
Their CIO challenged his team to be more responsive to marketplace dynamics by bringing features and functionality to the market faster, while driving cost efficiencies by scaling down virtual machines (VMs) and optimizing infrastructure.
The short version of this story is that this customer, who achieved over 10 billion USD in revenue in 2021, worked with Capstone IT and VMware Tanzu to adopt agile practices and a Kubernetes platform that could optimize their IT infrastructure for increased profitability. Capstone trained over 100 of their staff in Kubernetes and helped them move from monthly to daily deployments while reducing hundreds of VMs and CPU cores. The customer is now anticipating large savings with a payback over approximately five years.
If you are interested in learning more about where this initiative started, as well as how they achieved their goals, keep reading.
The technical debt curve: shifting spend from debt to new profitability bets
Overall, this story is about an IT organization trying to become more efficient. They reached a point where IT spending was growing faster than the company’s revenue and the CFO needed to flatten the spending curve for a sustainable operation. They needed to ensure that the IT organization was contributing to profitability instead of increasing expenses and, for that, they were looking to modernize a few internal practices.
Capstone was the partner for this project and was a great fit since they have been “doing containers before containers were cool” by partnering with the largest vendors in the industry for more than seven years.
Capstone realized that this customer’s IT organization had extensive planning cycles before taking on projects and, even though they had an agile mentality, they also had high rigor in execution, which meant low flexibility (as they couldn’t miss or tweak much on the go).
Their infrastructure was operated on-premises with no public cloud consumption, and tickets triggered infrastructure requests from the development team. This led to some friction, since time to resolution was a perceived challenge. In short, their organization couldn’t deliver fast enough.
This infrastructure team was measured on cost efficiency (what is the cost of a VM?) and had a very expensive environment to run, which included duplicative infrastructure for test environments as they wanted to run various tests, impacting optimization.
The plan for modernization that Capstone wanted to pursue was to have multiple parts of the organization running their own agile flywheel/practice independently.
Doubling down on agile practices and containers
To help this customer’s IT organization be more programmatic and ensure they were of service to the development organization and not an impediment, Capstone needed to remove silos (e.g., each team using their own JIRA boards), which also helped the customer become more collaborative and speed up their path to production.
Before containers, the development team would go into a monthly flywheel/practice of building and reloading, inspecting, committing, and writing the code. They would buy new hardware, size new storage, as well as a myriad of other things in a highly methodical approach.
The customer needed to move their production cycle from months to weeks and adopt containers. This took some convincing and education, which was to be expected coming from a traditional organization.
Visualization of the path to production cycles before and after Kubernetes
But even though the customer had traditional teams for infrastructure and applications, they also started a DevOps practice which was an initial effort to improve their pipeline. So, there was a clear desire to move to a more efficient state.
The customer prioritized three applications for that modernization journey:
- Critical business-to-business – For interaction with resellers
- Business-to-consumer (requests) – To capture queries and requests on their platforms
- Business-to-consumer (support) – For sales support to a specific customer audience
Building a platform
This automotive company had a series of decisions to make when building its platform, and the first one was to offer developers pre-built clusters over clusters as a service. To get an idea of how this change might affect the developer or end user of a said cluster, consider that instead of a developer requesting a cluster and adding their various tools manually or via a script, they select a pre-built cluster type with the desired versions of applications or tools deployed for them.
As the customer was trying to deploy three big applications, including one that brought in approximately USD $1 billion in revenue, they decided that each app would get its own cluster in development, test, and production environments. By limiting each application to a dedicated cluster, they now have much more granular control over resource usage and can eliminate noisy-neighbor issues (i.e., if multiple apps are running in separate namespaces within the same cluster, one can potentially affect the other with high resource use). This was done by enabling vSphere with VMware Tanzu and creating workload clusters with customized CPU, memory, and disk configurations to suit their needs.
After that, Capstone used VMware Tanzu Mission Control to apply several policies, including:
- Access – Ability to give each development team access to their clusters and separate namespaces
- Network – Option to segregate apps and ensure they don’t receive unwanted traffic
- Security – Power to limit capabilities and privilege escalation within containers, with the ability to configure policies to monitor without enforcement, so violators can be identified
- Image registry – Opportunity to allow only known and trusted container registries to be pulled from, preventing images from untrusted sources avoiding external sources like Docker hub
The team also leveraged Kubernetes labels for two sets of information. With these labels, they are now able to track:
- Who owns what – Org A owns this specific resource, or those are all the resources owned by Org B.
- Step through the pipeline – Each step has a label so developers and operators can ensure that anything they deploy went through all steps necessary.
Supporting the developers in a smooth path to production
The developers in this automotive company needed Memcached and Redis to support their applications running in production; this is where VMware Application Catalog came in. It is a curated catalog of open source application components that are continuously maintained and verifiably tested for use in production environments, and privately delivered to customers as open virtual appliances (OVAs), container images, and Helm charts.
The infrastructure and the DevOps teams decided to utilize Harbor as their container registry and used its proxy cache capability to pull down container images and helm charts from VMware Application Catalog into Harbor.
To reduce errors in their continuous integration and continuous delivery (CI/CD) pipeline, Capstone wanted to make sure that the container image creation process on the developer's machine was the exact same in the pipeline, so VMware Tanzu Build Service was used for that.
The customer leveraged the auto dependency updater feature to automatically keep all container layers continually patched, which brings immense security benefits of not having to patch and upgrade operating systems, Java Runtime Environments, etc. Additionally, this saves time by eliminating the need to rebuild and patch all of the container images when remediating common vulnerabilities and exposures (CVEs).
So, this is how the CI/CD pipeline was working: the developer commits code, a pull request gets created, and gets approved. Jenkins picks up that trigger and runs any automated unit tests to hand off the container creation process to Tanzu Build Service, which builds the container image and puts it into Harbor.
Now, to optimize the CD side of the pipeline, the DevOps and Dev teams worked together to utilize Helm charts as their deployment method and used different labels to indicate the current code stage, as well as control the deployment of the Helm charts (e.g., management approved, QA approved, and InfoSec approved).
Considering that applications are now in production, the customer needed a service mesh and started using VMware Tanzu Service Mesh as an Istio lifecycle manager to enable mutual transport layer security (mTLS) between apps, in addition to mTLS to external services. In the future, the customer may opt for traffic shaping and/or leverage advanced features such as the global namespace and service level objective.
The final step was to add Aria Operations for Applications (formerly known as Tanzu Observability), a full-stack solution for observability and logging that allowed the development team to see their applications running in production with metrics they have never seen before (e.g., pod restarts, CPU utilization, error rates, requests in and out of apps, etc.). But, most importantly, the customer was able to use their new overview dashboard to see how much more efficient this process was, and use that as motivation to help other business groups transform.
Roles and responsibilities for Aria Operations for Applications and all the Tanzu solutions used in this modernization effort.
Capstone and VMware: one team, one outcome
One of the DevOps engineers at this automotive company said that a massive outcome was the ability to push deployments during the day. And to achieve such a huge transformation, it certainly takes a village.
To conquer the challenges our customer was facing, many solutions and capabilities were needed—thankfully, due to a great partnership between Capstone and VMware, coupled with the solutions offered by Tanzu for Kubernetes Operations and Tanzu Application Platform they were able to meet these challenges head on.
Capstone created a brand new cluster on vSphere to offer a new private cloud, trained over 100 people from different groups on Kubernetes, and established a container environment to improve resource utilization through containerization to ultimately improve speed and stability through application migration.
Capstone also did some extensive financial calculations distilling everything down to the price of a VM (i.e, licensing, cables, boxes, servers, etc.) to show the retailer the flattened curve they needed for the best return on investment.
If you are wondering what it took for this team to be successful, it took three (not-so-simple) steps:
- Met the customer where they were – Understanding the current environment, processes, and culture intimately
- Solved the business problem – Not a product or particular technology problem while leveraging the entire Tanzu portfolio and resources
- Communicated in the spirit of business outcomes – Fiscal outcomes in addition to cultural and organizational outcomes
And the key results they delivered to this retailer were:
- Daily deployments
- Patching cycle improvements resulting in great return on investment
- Reduction of hundreds of VMs and CPU cores
- Dramatic resource management improvement expecting to move from 20 percent to 85 percent in infrastructure utilization on the path to complete migration from legacy to the new environment
- Anticipated savings of hundreds of thousands of dollars over five years
For more details on this story, watch this VMware Explore session. To learn more about their services, reach out to Capstone today!
