apache_hadoop events pov

Paul Maritz at Structure: Hadoop is Just One Ingredient of a ‘Profound Shift’ in Software

 

Paul Maritz at Giga OM Structure Data ConferencePivotal’s CEO Paul Maritz sat down with GigaOM’s Om Malik today at the Structure Data Conference in New York. He started the session picking up on a new trend that he is seeing in the market today, “For the first time you are seeing old line industries from the industrial space to agriculture to medicine, saying we need to build a whole new capability, and we are not going to try and do that on top of our legacy. We are going to build a new platform, and the legacy has to cleave to that, not the other way around.”

Maritz is not just talking about Big Data, the subject of the conference today. In fact, he calls Big Data technology Hadoop out as a catalyst to the market, citing that the bigger trend is in software is to take lots of cheap machines and cheap storage, and reinvent how businesses are building applications. Businesses, like large telecom companies whose businesses have matured to the point where they’ve managed to sell a phone to “everyone who can fog a mirror”, are forced today to pay close attention to customer satisfaction in order to remain competitive. To do this, they need to get down to the individual level at the moment, and respond intelligently.

“Most telcos have no way to really tell the actual experience an individual consumer has of their product…They treat everyone the same. They need to know how to differentiate in real-time and be able to understand different experiences,” explained Maritz. “They have to take the offline profile data they’ve built up of their customers and intersect it with high-speed, real-time data coming off their backbones and take decisions in a matter of seconds to decide what they want to do. That is now going to become a matter of life and death for those industries, but there is nothing within their existing IT systems that can do that.”

This is the trend Maritz saw to build Pivotal a year ago. “We believed that we needed to put together the right set of ingredients that would allow mere mortals and enterprises to be able to do what the consumer internet giants have done, and exploit the two quantities that have really driven their capabilities, which is the paradigm of using lots and lots of cheap machines working in parallel and lots and lots of cheap storage. And if you abstract far enough away from Hadoop, you realize that Hadoop is really an instance of that paradigm. It’s a framework for exploiting lots and lots of cheap machines and lots of lots of cheap storage. That paradigm needs to be extended quite radically, beyond the somewhat idiosyncratic way of working with MapReduce into other semantic paradigms of working with information. Secondly, we need to find ways for people to build these applications that can exploit the resources of the cloud, and can exploit who can give them the right cost structure of lots of cheap machines and cheap storage in the right location.”

The possibilities are far reaching, but it requires a mindset shift. For old industries, it requires leadership with clear enough vision to respond to the opportunities of data centric applications and services. For many, that vision doesn’t have to be even that far reaching. As Maritz quips, “there is nothing like the sight of the gallows to respond to threat.”

New companies like Uber and Lyft are emerging that take a very data centric point of view towards car services. They’ve set up cheap infrastructures, provided great user experiences from ordering the ride on the phone to the car ride itself, and are disrupting the traditional taxi industry. To survive, taxi companies are going to need to embrace the new deal. To catch up, they are going to need a platform, like Pivotal One, that will afford them the speed they need to catch up and allowing them complete latitude to invent new competitive edges.

cta-pivotalHD

This is why companies like GE famously embraced the Internet of Things, so they could reinvent how they provide serious competitive advantages to aviation, rail, and energy companies. While they used telemetry and big data technologies to solve this, they used development tools that went beyond just Hadoop. They used a new platform. And their results are remarkable. In just 6 months, GE innovated 14 new technologies that are projected to boost $10-15 trillion GDP by 2020.

It is why old agriculture companies are bringing data, telemetry and computer programs literally out into the fields. They are realizing that if they can use weather maps, soil sample data and planting grids, they can actually plan how to plant seeds that can yield up to 10% more crops. Double digit growth is attractive in any industry, and virtually unheard of in one as big as agriculture.

With the next set of great companies emerging with great results, Maritz sees that businesses are starting to wake up to the idea that they need to get onto a new journey: one that depends on a new platform for this new era of great software. The good news is, as Maritz sees,”The trajectory is set, and its going to happen, but we are in the first third of that journey.” So there’s time for companies to still move, but its getting shorter every day.