From the early days of VMware, it became clear that dealing with storage in efficient and scalable ways were key requirements for the success of virtualization in enterprise environments. Indeed, the storage stack of ESXi, including VMFS, played a key role in the proliferation of virtualization in data centers, where data is stored and managed by high-end disk arrays.Based on those foundations of storage virtualization, over the years, VMware introduced a range of availability and data management solutions ranging from HA, DRS and FT to Disaster Recovery and Data Protection. Moreover, VMware encouraged and supported a vast ecosystem of partners who innovated on the vSphere platform. See for example, the VADP ecosystem and more recently Virtual Volumes and VAIO I/O Filters.
But the world of technology is not standing still. For one, the increasing power of low-cost servers is making it possible to replace specialized hardware with general-purpose servers plus software. New, scale-out software products that run on commodity hardware are displacing traditional “monolithic” storage arrays.
This trend is accelerated by three factors:
- The emergence of new storage technologies, NAND-based Flash and soon-to-arrive Non-volatile memory. The low latency and high throughput of those devices mean that they should stay close to the compute nodes where the workload runs.
- New use cases for big data analytics. Their scale and performance requirements far exceed the capabilities of traditional storage products. Unfortunately, the result is the development of use-case specific storage solutions and thus the proliferation of “Storage islands” in IT environments.
- Application design is changing rapidly towards large-scale, micro-service based architectures. The storage and data management requirements of these applications cannot be met by the traditional storage arrays (whether block or file).
It was in this context, that five years ago (almost to the day), VMware decided to develop a new-generation storage platform. A platform that is natively integrated with the core features of our hypervisor (ESXi) and the management tools and services built around it. A platform that can be morphed into different storage and data management products that serve the emerging needs of old and new customers alike. That was the genesis of VMware’s Virtual SAN (VSAN).
Since then, I am very fortunate to have been working with a brilliant, highly motivated team, who developed and shipped VSAN in a very fast pace while adhering to VMware’s high quality standards. While at that, we made huge strides in a number of areas:
- We changed internal development and testing processes to better position VMware’s R&D for the Cloud era.
- We started shifting the architecture of our management stack and tools towards a more modular and scalable architecture.
- Last but not least, we opened up the control plane of our platform to our partners, with Storage Policy-Based Management (SPBM) and VASA. That’s the same control plane that we use for VSAN. Earlier this year, we shipped Virtual Volumes (vVols) based on that work.
The VSAN technology has been packaged and is shipping as a Hyper-Converged Infrastructure (HCI) offering in vSphere, since March 2014. Product awareness and customer adoption has exceeded our wildest expectations. In the first 9 months, we had over 1000 customers and the number is growing fast. Many of these customers are using the product in demanding production environments (for example, see the customer stories on Duncan Epping’s blog site). Innovation on VSAN will continue at a fast pace. Features, both core and management, will be released at a cadence much more aggressive than the rest of vSphere.
At this point, VSAN is targeted as a product for the storage of virtual machines and their virtual SCSI disks. Management is geared towards ease of use for IT professionals who are not storage experts, while still providing unparalleled visibility and troubleshooting capabilities when needed. For example, see the VSAN Health Check tool. With vROps Management Pack for VSAN, we are moving towards a highly scalable infrastructure for monitoring and analytics.
This is a good starting point, but our goals are even more ambitious and forward-looking. First, new applications and use cases are looking for data protocols and management services that go beyond virtual SCSI devices and beyond traditional vSphere clusters. Take the example of cloud-native applications (CNA), which are deployed using containers (e.g., Docker) on infrastructures that span 100s or even 1000s of hosts. CNA assume a file interface to storage, for OS images, configuration and even persistent application state. VMware is heavily investing in these new areas with a number of highly visible products and community projects (see http://blogs.vmware.com/cloudnative).
Second, customers have been asking for comprehensive data protection solutions beyond today’s point solutions for backup and disaster recovery. They request powerful data services to perform efficient manipulation and transformation of large data sets (see also http://blogs.vmware.com/cto/balkanization). VMware offers a range of successful data protection solutions today, including Site Recovery Manager, vSphere Replication and VADP. Based on that background, we are building next-gen data management solutions that work with any storage platform, not just VSAN. They will cover not only on-premise deployments across multiple data centers, but they will also enable a real hybrid Cloud experience for VMware customers and partners.
Recognizing the huge opportunities for storage and data management that lay ahead, VMware established a new Business Unit focusing on Storage and Availability, headed by Charles Fan (SVP and GM). Now, we are creating a new Office of the CTO in the BU and I am thrilled to be heading it as the CTO for Storage and Availability. The goals of the office and me personally are twofold:
- Define VMware’s vision and strategy and communicate it to customers and partners.
- Oversee the technical investments and directions of the BU and ensure that we are executing on the strategy we outlined.
I am very fortunate to be joined by two seasoned professionals who bring a lot of experience and credibility on the table and who share my passion and vision for technology and for the directions of the industry. Effective immediately, Duncan Epping (http://www.yellow-bricks.com) and Rawlinson Rivera (http://www.punchingclouds.com) will join the new OCTO reporting to me. This is a team of rock stars! I look forward to working with them. And I look forward to talking to all of you about VMware’s products and our vision about storage and availability.
