Customer Challenge
2020-2030 is the decade of data. The amount of data is growing exponentially (175 zetabytes by 2025), and the variety of data applications is also growing exponentially. At the heart of all these modern data applications is a storage foundation. Each data application has different access patterns and expectations (cost, SLAs, etc.) from the underlying storage system.
We have created a disaggregated cloud storage and data management solution that is able to adapt to the different expectations from the continuously evolving modern data stack.
Our Vision
Our vision for VMware Cloud Flex Storage is to deliver an enterprise-class storage-and data management-as-a-service for the multi-cloud. We plan to support a broad range of workloads by enabling multi-dimensional scaling of compute, storage performance, and storage capacity, for both traditional and modern workloads. Over time, we plan to extend the service to multiple clouds and deliver a seamless and consistent data management experience across clouds.
VMware Cloud Flex Storage is built on a mature, enterprise-class file system that has been developed and production-hardened over many years, and also backs VMware Cloud Disaster Recovery. The two-tier design of VMware Cloud Flex Storage enables independent scaling of storage performance and capacity. The combination of the filesystem’s two-tier design, Log Structured File System design, app-centric snapshots, instant cloning, and immutability, makes it a multipurpose file system. With VMware Cloud Flex Storage, we will extend this proven technology to cloud storage, where it delivers exceptional storage performance, scalability, and cost efficiency.
We will describe a few architectural design elements of VMware Cloud Flex Storage below. Note that a subset of these features will be available in the first release.
2-tier Design
VMware Cloud Flex Storage uses a 2-tier design: a capacity tier, and a performance tier. All data is stored in S3 (capacity-tier), and we use EC2 with NVMe for IO performance (cache-tier). This 2-tier design helps us decouple storage capacity from storage performance and be able to dynamically scale them independently. This flexibility allows users to balance cost and performance based on the needs of their data applications. For example, OLTP databases need very high random IO performance, OLAP databases need high sequential IO performance, and file shares have a mix of 20%/80% hot/cold data.
Efficiencies
VMware Cloud Flex Storage use a Log-Structured Filesystem (LFS) layout design. The original idea for LFS was first proposed in 1992 by Mendel Rosenblum, who also happens to be the founder of VMware. The crux of the idea is to append incoming data into a sequential log and do the garbage collection later. Most raw devices (HDD, SSD, S3) are not good at random writes, but all are very good at sequential writes. The LFS design allows us to provide a flexible multi-purpose filesystem.
We employ the LFS techniques to store data in S3. As shown in Figure 1, all incoming data is converted to large ~10MB sequential segments, and these large segments are stored as S3 objects, and S3 is excellent at large sequential IOs. This allows data to be stored in S3 at high-speed. The LFS design also makes it easier to implement other cost efficiencies (dedupe, compression, thin-provisioning), and ultra-efficient snapshots and clones.
Instant App-centric Snapshots & Clones
We live an instant economy, and we now expect our storage systems to also provide instant data management operations. VMware Cloud Flex Storage uses a variety to techniques (like LSM-Trees) to provide the ability to create snapshots instantaneously. Instead of being a dumb volume level snapshot, we have built mechanisms to create app-centric snapshots (simple apps and complex distributed apps). These snapshots are highly efficient in terms of cost and space usage and can be used to instantly create clones of applications. Instant snapshots and clones are useful for many data management workflows: operational recovery, CI/CD data pipeline, test/dev, etc.
Availability of VMware Cloud Flex Storage
Today, we are excited to announce that VMware Cloud Flex Storage will be available in VMware’s Q3 FY23.
In this initial release, we will deliver a new approach to help VMware Cloud on AWS customers better align their cloud resources with the needs of their applications and data. Customers will be able to purchase a disaggregated cloud storage and data management service that is fully managed by VMware. It is scalable, elastic, and natively integrated into VMware Cloud on AWS. With just a few clicks in the VMware Cloud Console, customers can scale their storage environment without adding hosts, and elastically adjust their storage capacity up or down as needed for every application.
VMware Cloud Flex Storage will be offered as supplemental storage to vSAN for capacity-heavy workloads. Together with vSAN, VMware Cloud Flex Storage will offer more flexibility and customer value in terms of resilience, performance, scale, and cost in the cloud.
“VMware Cloud Flex Storage brings the flexibility needed to design VMC solutions to my customers. Having the choice in storage platforms will help drive customer adoption of VMware Cloud on AWS for additional use cases and helps unlock the value of the platform even further.”
Ken Nalbone
Senior Solution Architect, AHEAD
We will offer VMware Cloud Flex Storage via a simple, pay-as-you-consume $ per GiB pricing model. There will be on-demand, 1-year and 3-year subscription options. The solution does not require pre-provisioning of storage increments, which provides more flexibility and granular consumption. The simple, pay-per-utilized GB consumption model means there’s just one price and metric, allowing you to forecast and plan your spending with greater accuracy.
How to Learn More
Please visit the VMware Cloud Flex Storage page available here and contact your sales representative or partner for more information on this service and how VMware Cloud Flex Storage can help your business.
Disclaimer: VMware makes no guarantee that services announced in this announcement will become available at a future date. The information is for informational purposes only and may not be incorporated into any contract.
