Statements that “we’re committed to open source software” are easily made by technology vendors, and are probably just as often dismissed by their skeptical audiences. Why and how are they committed? What exactly does open source software mean to that vendor, and how does it relate to their commercial efforts? Different vendors will have different answers to these questions according to their offerings, and would-be users, customers and employees are wise to inquire into them. In this post I want to explain Pivotal’s perspective on open source, what a commitment to it means to us, and why we think you should care.
To understand our relationship to open source, it’s important to keep in mind what we’re trying to do, and for whom. Pivotal’s customer is, in a word, the enterprise. And our mission is enabling them to radically improve the way they build, scale and analyze their applications.
When we talk about this mission, you’ll often hear us say that we want to help enterprises think and behave more like Google and Twitter when it comes to working with massive amounts of data and building applications that users love. There is no technical reason why enterprises can’t do this, and there is every business reason why they must. Software is indeed eating the world, and all companies—all enterprises—will be judged on how they adapt their products and services accordingly.
Take insurance for example. Nothing about the Internet has fundamentally changed the business of risk management, but the ability to file a claim from a smartphone at the scene of an accident may be the difference between keeping and losing a tech-savvy customer. This is only more true when your competition is advertising that ability directly to that growing demographic.
This developer would typically hesitate to quote an analyst, but Gartner are spot on in their assertion that every company is a technology company now. The ones who don’t yet recognize themselves as such will just have that much harder a time competing. The ones who do should want nothing more than to face necessary changes head-on and get ahead as a reward.
You can think of Pivotal as a catalyst designed to accelerate those ‘necessary changes’, and we believe that open source technology is integral at every step of the process. While a comprehensive list is a matter for another post, here are two such changes that most enterprises will need to move through in order to realize their full technology potential:
1. Focus on technical talent. For enterprises that want to compete, outsourcing tech work is over (and it never worked well anyway). Think about it: Google, Facebook, Twitter, LinkedIn, Airbnb and the like have defined the modern application user experience by attracting and keeping top talent. Ambitious enterprises should follow suit, and therefore prepare to insource. This will be hard to do without embracing open platforms and technologies, as the vast majority of developers prefer working with them. Just ask our team at Pivotal Labs, who for years have been helping clients build world-class apps with open source software, and who make it their business to teach those companies how they do it along the way.
2. Favor technology stacks that aim for openness, integration and pragmatism. This much is clear: the era of the proprietary single-vendor stack is over. One size does not fit all, and one knows this is true when even Microsoft’s .NET ecosystem has embraced open source. Open source allows enterprises maximum freedom to do what they need when they need, while at the same time maximizing developer engagement. Openness alone is not enough, however. The pieces of a technology stack need to hang together with a cohesive programming model and common APIs whenever possible. This kind of integration maximizes productivity and allows developers to move between different aspects of an enterprise system with minimum overhead. And pragmatism means understanding that sometimes the best choice is to build instead of buy; that sometimes the best technology for the job will be closed-source, not open; and that the most important goal should always be to deliver. We’ve kept each of these values in mind when designing Pivotal as a company, and when designing PivotalOne as a platform.
With our mission in mind and a sense of what it means to accomplish it, we can answer the three questions that began this article.
Q: Why is Pivotal committed to open source?
A: Because there is no way we could accomplish our mission without it.
The story of the last decade or so of enterprise application development could be titled The Triumph of Open Source. In the early 2000’s, teams were still debating whether open source web frameworks could really compete with home-grown ones, whether open source databases could perform on par with their proprietary competition, and so on. In 2014 one would likely be laughed out of the room for asking such questions. Developers steadily fought and won the battle for open source supremacy in the enterprise, and we want to build on that momentum. Pivotal and our products are relevant because we operate with the assumption that teams want to deal with open source technology as much as possible, and resort to closed-source products only when the circumstances justify it. We think we’ve got the balance right in our product portfolio, but our customers will be the final judge.
Q: How are we committed?
A: Our commitment to open source is manifest in three ways:
1. Many of our most important offerings are themselves open source projects (see below for examples).
2. We employ committers for those projects (the majority of committers in many cases).
3. We participate in and/or lead standards bodies, foundations and expert groups where applicable
Q: How does open source relate to our commercial efforts?
A: Our open- and closed-source products exist by design in symbiosis with one another.
In general, licensing fees from our closed-source products help fund our open source work, while our large open source communities help expand the potential audience for our commercial products and open source support contracts. Generalities aside, however, the best way to understand this often nuanced relationship is to look at each product in turn:
Cloud Foundry
Cloud Foundry was initiated as the industry’s Open Platform-as-a-Service (PaaS) to be the Linux of the Cloud. Open PaaS not only simplifies application deployment, scaling and operations but also provides developers’ choice of frameworks, services and clouds. Open PaaS is bigger than any group of individuals or organization and that’s why Open Source Cloud Foundry has become the channel for broad industry participation in shaping the Platform for the cloud era. In the last year alone, we’ve had CenturyLink/Savvis, IBM, NTT, Orange, Rakuten, SAP, Symantec, SwissCom, Verizon and many others become contributing members of the Cloud Foundry community.
Hadoop, Pivotal HD, and HAWQ
Hadoop is the branding iron that seared the words “big data” into the IT industry. As an open source project for analyzing large sets on low-cost hardware, it has brought a disruptive change to the cost of analyzing massive amounts of data, becoming notorious and pervasive. Pivotal, like other vendors, includes the core, open source components in our own distribution. We have built a team of people to contribute to the Hadoop open source project, provide support, and develop an integrated set of licensed software that sits alongside and adds significant value to the Hadoop stack.
Spring, Groovy and Grails
Like most big web players, the majority of enterprises today are “Java shops”. The JVM remains formidable in its ubiquity, stability and performance characteristics, while the larger Java ecosystem is rich with capable developers and mature open source libraries. For years, Spring’s mission has been to provide comprehensive infrastructural support for enterprise Java applications, and that mission continues stronger than ever today at Pivotal. Spring is the programming model that provides elegant, unified access to nearly every other technology in the PivotalOne stack. Spring provides developers with a cohesive, familiar and flexible approach to building even the most ambitious applications. Spring is particularly important to Pivotal’s strategy as its wide industry adoption means that Pivotal has immediate access to development teams everywhere.
Spring Data GemFire and Spring Data Redis
Spring Data projects are shining examples of how Spring projects act as a unifier across software of all types—both open and closed source. It provides several sub-projects that are not datasource-specific like Spring Data JPA, Spring Data REST, and Spring Data JDBC Extensions. Spring Data also includes projects by some of the companies behind newer, more popular data stores. These projects make it easier for Spring-based applications to access their data services. For example, there are projects for MongoDB, Neo4J, Apache Solr, Couchbase, Elasticsearch, and Hadoop. In addition, Pivotal has contributed Spring Data Redis and Spring Data GemFire for easier configuration and access by Spring applications.
Redis
Another open source project that has transformed big, fast data into a reality, this software is used by companies like Twitter, Pinterest, Hulu, and Viacom to manage billions of key-value pairs. Started in 2009, Redis inventor Salvatore Sanfillipo is now part of the Pivotal team and getting support to make sure the project evolves as fast as the internet giants deploying it. As the largest key-value store used on the planet, it is important to us to invest in its future, ensuring it can be easily deployed as a key part of application’s data fabric, even offering commercial support for both developers and production systems.
RabbitMQ
Pivotal employs the main engineering team behind RabbitMQ, the leading open source message server with an estimated 50,000 production sites. Built on the AMQP open standard, it is the secret to speed for internet giants like Instagram, Huffington Post, and Soundcloud. An industry standard, you’ll also find it bundled in Chef, Fedora, Debian, Ubuntu, vCloud Director, OpenStack, and others. Offering support to connect components across Java, Scala, Clojure, PHP, C++, Go, COBOL, OpenVMS, Python, .NET, Ruby, and more, RabbitMQ is the universal connector for loosely coupled systems. It is important we keep it open to ensure maximum flexibility to evolve as new technologies are incorporated into application fabrics.
Apache Tomcat and Apache Httpd
Proprietary application and web servers gained a reputation long ago to be not only bloated, but a gate to scaling your business on the web. Lightweight, open source options gained enormous traction over the past decade, with Apache leading the pack since 1996. Given our position that Java is the future for scalable, enterprise applications, it makes complete sense that Pivotal contributes heavily to the leading Java application container and web server, including contributing dedicated engineers to work within the Apache Software Foundation and function as release managers, contributors, and lead focused efforts around security and running their bug tracker.
MADLib
One of our most recent open source projects, MADLib started off as collaboration between database engine developers, data scientists, and academics, based off of a 2009 VLDB paper. Our investment in this project is justified simply by the fact we believe scalable, distributed, in-database analytics on big data (in the terabyte and petabyte range) is becoming critical to the survival of any enterprise. The toolset includes mathematical, statistical, and machine learning methods for structured and unstructured data that offer data scientists a proven head start on their projects, reducing the risk and cost to start up their next big data project, which is a win-win for everyone, especially their customers.
Conclusion
For some vendors, a commitment to open source might mean improving technology while cutting costs; for others it’s about giving back; and for others still open source is a way to attract great talent. We applaud it all; everybody wins. For Pivotal, however, the commitment goes even deeper—we’re banking on open source in nearly everything we do. And that’s why, for us, open source is Pivotal.
