The Drive Towards Cloud
This blog showcases VMware’s vision for the future of data, data management and storage. There is no commitment or obligation that the features detailed below will become generally available.
The drive of the IT industry towards cloud is making headlines almost daily, with most analysts focusing on public cloud providers. Indeed, the combined market cap of Amazon, Microsoft and Google, the top three public cloud providers, is approaching six trillion dollars – almost one-third of the entire Nasdaq capitalization.1
The public cloud leads with simplicity, elasticity, a quick time to value and perhaps most importantly, a low barrier to entry, so it makes it easy to try new projects. There is an incredible opportunity for private data centers to deliver similar value, and recapture hundreds of billions of market share from public clouds. Andreesen Horowitz eloquently articulated this opportunity in their thought piece “The Cost of Cloud, a Trillion Dollar Paradox.”2 VMware’s unique value is the ability to span across both private- (data center and edge deployments) and public clouds – which illustrates the big opportunity it has for a Federated Storage Platform.
In this blog, we describe what this means for VMware’s strategy for data, data management and storage.
Storage Challenges
Traditional storage arrays are defined by fixed hardware boundaries, which confine their resources. As a result, they are hampered by trapped data, stranded capacity, limited scalability and elasticity, complex placement decisions, limited data and compute mobility across (physical or logical) system boundaries, and high sunrise and sunset cost. They require upfront obsolescence planning with significant investments to placate technology risk, conflated by disruptive and expensive end-of-life tech forklifts with data migration sprees. In short, traditional storage array architectures cannot deliver a cloud-like experience. The industry’s continued march towards the multi-cloud/zettabyte era is only exacerbating these limitations.
Hyperconverged Infrastructure (HCI) has successfully stretched system boundaries and created a whole new industry. It provides x86 server economics, combined with software-defined infrastructure simplicity and flexibility. VMware HCI Mesh expanded HCI’s flexibility through compute and storage disaggregation, and HCI serves as a foundation for hybrid cloud.
Now, VMware is continuing to expand its thinking to tailor its software platform for multi-cloud use. Storage is foundational because every business is a data business, so we’ll focus on that first.
A New Storage Concept for the Multi-Cloud Era
We’re introducing a new concept: the Federated Storage Platform (FSP). Our vision is that the FSP would orchestrate data across the data center: across any number of vCenters and storage backends. The FSP would provide seamless compute/storage disaggregation, data center-wide data accessibility, data center-wide storage policy based management according to application needs (including global placement), data center-wide storage and data insights, data center-wide VM migration and seamless storage expansion.
As a result, the Federated Storage Platform would be able to drive cloud-like storage operations and deliver cloud-like self-service storage consumption based on application needs.
How Does it Do That?
FSP would converge mixed storage systems into one single cloud-like consumption surface that could span the entire data center:
- Common volumes: A new presentation layer concept that converges mixed physical storage resources from heterogeneous vSAN clusters, VMware vSphere Virtual Volumes storage arrays and NFS storage arrays into a uniform data infrastructure, cleanly abstracted by the platform.
- Data center control plane: A new control plane concept that composes the data infrastructure into elastic pools with full visibility into how it’s consumed. This would provide a cloud-like infrastructure consumption experience across heterogeneous storage types, and a common surface to mesh with other services.
A Few Key Benefits
Data center-wide coherent storage policy
FSP could be fully integrated with SPBM (vSphere Storage Policy Based Management), so that the data center control plane automatically manages storage pools and policies, based on application needs.
vSphere consumers could define these policies to meet their requirements, and the data center control plane could determine which of its storage types meets them best, instantiates the storage and places the data accordingly.
Holistic insights
The FSP could support advanced policies like capacity quotas or performance limits, for instance, to enable the support of multiple service tiers.
A goal for the FSP would be to converge physical storage silos across the data center into one single global view, including all provisioned objects, policies, and so on. With its single global view, FSP would have full visibility into all consumers of all of its datastores exported across all vCenters. Therefore, it could significantly extend the current management capabilities of vSphere with much broader, more holistic resource decisions.
Seamless storage expansion
The FSP could seamlessly add new storage. For instance, new storage can be added without requiring the replacement of current storage, eliminating disruptive forklift upgrades.
In this scenario, the data center control plane, would automatically discover and add the new storage to the federated store. It could also load-balance to new storage by automatically orchestrating VMware vSphere Storage vMotion. This de facto elimination of silos promises advantages like eliminating hot spots, mitigating capacity pressure, and/or avoiding noisy neighbors. Further, the data center control plane would only need to move the storage, not the compute, since any compute could have access to any storage via Common Volumes. Even better, it would automate the entire process on behalf of the administrators and app teams.
In contrast, a VI Admin doing this today needs to schedule Storage vMotion operations manually, which is error-prone and disruptive. The FSP would “lazily” move and rebalance data objects in the background by driving Storage vMotions, and inform the VI Admin as equilibrium is approached or achieved. This also includes storage retirement: FSP could retire clusters that have reached their end of life by orchestrating Storage vMotion to evacuate and decommission them, all with selectable degrees of automation.
As a result, FSP would continuously roll storage beneath a stable cloud consumption surface, essentially providing perpetual storage that could dynamically and constantly evolve, based on what applications and customers want. The FSP Concept would thus deliver on our vision of a cloud-like storage experience.
Integration with CSI and CNS: Cloud-scale storage for modern applications
FSP would be integrated with the vSphere CSI driver in Kubernetes, as well as CNS (Cloud Native Storage). Developers can continue to consume vSphere storage through storage classes in Kubernetes, just like they do today, but FSP would now give VI Admins full visibility into how Kubernetes applications are consuming vSphere storage across the whole of the data center.
VMware vSphere with VMware Tanzu and TKG-S VMware Tanzu Kubernetes Grid Service workloads can currently provision workloads on storage that is accessible to the cluster where the namespace is provisioned. FSP would extend that accessibility to enable these workloads to access any storage in any cluster in the data center with any policy.
Seamless workload migration
Today, compute (VMs, containers) can be moved between vSphere infrastructures together with their entire state (including storage) through x-vMotion.
FSP would reduce or eliminate the impact of vCenter- or cluster boundaries by enabling compute (VMs, containers) to freely float across the boundaries of its corresponding vSphere infrastructure. It could enable compute to access storage from any point in the data center, so compute could easily move between vSphere clusters, data centers or even vCenter Server instances.
Scale different
Customers may continue to have multiple vCenter servers for organizational or geographical reasons but customers are also looking to efficiently deliver a consistent, seamless, simple, cloud-like experience that can be managed centrally across distributed environment.
Our vision of a Federated Storage Platform would provide a single cloud-like consumption surface for disaggregated, mixed storage for vSphere infrastructure across the entire data center.
This makes the FSP concept a great step towards realizing VMware’s vision of seamlessly distributing applications across multiple clouds.
Learn more in our VMworld 2021 session: VMware’s vision for storage and data in a multi-cloud world.
1. VMware Internal Analysis.
Sources: “Largest American Companies by Market Capitalization.” CompaniesMarketCap. 28 Sep 2021. https://companiesmarketcap.com/usa/largest-companies-in-the-usa-by-market-cap/. Accessed 28 Sep 2021.
“Nasdaq.” Wikipedia. 28 Sep 2021. https://en.wikipedia.org/wiki/Nasdaq. Accessed 28 Sep 2021.
2. Casado, Martin and Sarah Wang. Andreessen Horowitz. “The Cost of Cloud, a Trillion Dollar Paradox.” https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/. Accessed 28 Sep 2021.
