Home > Blogs > VMware vFabric Blog


2 Ways Fast/Big Data Impacts the IT Org Structure

Though my background includes time as both a developer, architect, and CTO, much of my time today is spent discussing applications with senior IT executives. I manage an application development division of a national VAR and focus on the vFabric stack from top to bottom. One of the challenges I face is trying Screen shot 2012-08-08 at 10.48.39 AM
to provide application-centric consulting services to operations/infrastructure teams who (a) don’t really own the decision of app software infrastructure and/or (b) don’t understand it and, (c) worse in some cases, don’t care. Recently, I’ve come to love my job for two primary reasons:

1. “Cloud” technologies are forcing the Operations teams and the Application teams to “share” responsibility for overall IT efficiency. The cloud concept of an on-demand, elastic infrastructure is knocking down political walls and silos that have evolved over the past decades in IT. This is no more evident than at VMWare, where vFabric and vSphere product lines are starting to blur (e.g. vCenter –> vCloud Director –> Application Director). Finally, I have something to talk to the Infrastructure folks that gets them excited! Perhaps it is the needed automation of infrastructure that brings Ops to the Aps side. Or, perhaps it an elastic architecture that brings Aps over to the Ops side. In any event, the two teams are brought together and work together more in cloud solutions.

2. Fast and Big Data systems (a.k.a. transactional and analytical scale-out distributed databases) are at the heart of what is driving this convergence. Ironically, DBA’s often report in to the infrastructure groups. One of the main reasons is RDBMS systems become monolithic and resistant to change – DBAs have very little to do on the application side and almost everything to do on the operations side (especially disaster recovery and high availability). For example, most DBA’s know a lot about SAN replication. I know very few software devs who do. With the advent of Fast Data NoSQL systems like SQLFire/Gemfire, MongoDB, Cassandra, HBase, etc and Big Data analytical systems such as Greenplum, Hadoop/HDFS, Cloudera, Hana, Vertica, etc, the walls between IT Operations and Applications are being forced down. The developers using these systems today are stepping back in to the domain of DBA’s. And, they are modernizing their applications to fit these new, scale-out architectures. By doing so, they create efficiencies in IT. For example, Ops no longer needs to worry about disaster recovery and fault tolerance and, instead, focuses more on the efficient use of hardware infrastructure.

Going to VMworld in Barcelona?

Register for VMworld!
Click Here

Register for Session (APP-CAP2757- Accelerate Adoption by Leveraging IaaS for a Complete Deployment and Monitoring Lifecycle):
Click Here

Register for Session (APP-CAP1250 - Fast Data Meets Big Data):
Click Here

Follow all vFabric updates at VMworld on Twitter:
Click Here

It might be an overstatement for me to liken this trend to the 1989 fall of the Berlin Wall, but, heck, for a guy like me and my job, it’s a sight for sore eyes. My audience no longer falls asleep during sales meetings : )

Fast/Big Data was invented for, or perhaps, because of the internet…it has to handle scale, both in terms of velocity of transactions and volume of data. And it is at the heart of the cloud trend. The RDBMS is the last stone to tumble from the old wall of IT. And as the application teams now stare through the dust and rubble at their operational counterparts, I think they should be jubilant because they now have something to build together — namely, a new architecture where high-availability, fault tolerance, and rapid scale-out are built in to the design. Where IaaS and PaaS become one. The database is the last thing standing in their way to accomplishing that feat. I look forward to the day when the Apps team won’t be able to throw some software over the wall to Ops and say “You worry about Disaster Recovery.” And, Ops won’t be able to point the finger at Apps and say “Your app doesn’t scale,” because they will be part of building the “cloud” architecture that unites virtualization with software development design in to a scale-out instead of scale-up model. This trend is rapidly changing the hardware VAR model; just look at the trend data on x86 shipments versus legacy Unix (and I dare say mainframe) hardware shipments. Traditional VAR’s will die if they rely upon the margins of X86 hardware; those that survive will need to provide value by moving up the software stack, past virtualization, past IaaS, and understand the actual workloads (such as Big Data) that are running on those machines.

While it’s only the early stages, I’m just happy to see that the train has left the station for some of my customers who are adopting Big/Fast Data solutions like SQLFire/Gemfire, Greenplum, and Cetas. And, it’s actually fun again to work in an environment where the Ops team and Apps team are both in the Engine room, stoking the fires and getting some real work done.

Jeff-reed-icon-from-LI_80x80 About the Author: Jeff Reed manages the application development division for Logicalis USA, where he also spent time managing channel relationships with major IT OEMs. He focuses his energy on modern application architectures built with vFabric and Spring technologies, and on the integration of IaaS (vCloud Director) and PaaS (CloudFoundry) platforms. Other than being a father to two amazing boys and owning a fly-fishing lodge in Montana, Jeff’s two proudest IT moments include being the lead developer for the original Outlook Web Access and serving as Chief Software Architect for T-Mobile  (Jeff may be reached at www.linkedin.com/in/jefftreed).

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>