Home > Blogs > VMware vCloud Blog > Monthly Archives: April 2010

Monthly Archives: April 2010

What Does DevOps Mean for Cloud Professionals?

By Steve Jin, VMware R&D

 

This
entry was reposted from
DoubleCloud.org, a blog for
architects and developers on virtualization and cloud computing.

 

I heard about DevOps a while back but didn’t really look into it. My
initial understanding was that the roles of developer and system administrator would
merge into one called DevOps. Last week, I attended a DevOps meet up in Palo Alto and got the chance to
learn from others about DevOps. The hosting organization even wrote up a good blog
defining what a DevOps is.

 

DevOps is, in many ways, an umbrella concept that refers to anything
that smoothes out the interaction between development and operations. However,
the ideas behind DevOps run much deeper than that. The DevOps is more about a
movement than the merging of two roles. The basic idea behind the DevOps is to breach
the wall between development and operations.

 

Traditionally developers ship products that are then run by operators in
other companies. In this new age where much of software is delivered as
services, the developers run their software directly. When there is a problem,
the developers must fix it – usually right away. That is why you see engineers
at Google required to rotate on calls for support. When more companies ship
software as services, it’s natural that more engineers will wear two hats on
their heads. The DevOps concept is not really new, but the terminology is.

 

With the DevOps movement, it doesn’t mean there won’t be a need for
system administrators. We still need system administrators who manage core infrastructure
such as storage, networking, and such, but fewer administrators to run and
monitor applications. Whoever designs and develops the software needs to take
care of the operations and support. As infrastructure gets bigger in scale,
companies continue to expect their system administrators to become good script
developers who can drive the most operational efficiency out of automation.

 

For the development side, there are even bigger impacts. For one thing,
you have to carry pagers over the weekends. Whenever there is a problem, you
have to fix it no matter how late it is. With this pressure on, all the
architects and developers need to think carefully about operations as they
develop software: how to make it stable? How to make it scale? How to prevent
system scope outage? How to quickly and silently recover from outage? How to
recover quickly without hurting SLAs? This change in traditional roles forces
them to come up with more scalable and reliable architecture, better quality
software, and easier to manage services. This is good for the people, good for
the companies, and good for customers!

 

Not every developer will be directly affected. We will still see
software as products in the future. But when enterprises move to private cloud
computing, more of these products will be architected to run as services or
applications on top of services. Having a

DevOps mindset and corresponding expertise will definitely be a big plus
for every architect, developer and system administrator. As a
cloud professional or would-be, are you ready for the challenges
and to embrace the changes?

 

Steve Jin is
author of 
VMware VI & vSphere SDK (Prentice Hall), founder of open source VI
Java API
, and is the chief blogger at DoubleCloud.org

Fog: Driving Clouds with Ruby

By Wesley Beary (aka
geemus), Engine Yard

 

Fog is the Ruby cloud
computing library. It
helps Ruby
developers interact with cloud services. Here is our logo:

VMware_fog

 So, the quick and dirty on fog, top to bottom:

* Models provide a simplified interface, making
clouds easier to work with and switch between;

* Requests allow power users to get the most out of
the features of each individual cloud;

* Mocks make testing and integrating a breeze.

 

As I was getting my head
wrapped around working with cloud resources and trying different services I
found myself wishing for more open, powerful tools. From this desire I started
working on basic wrappers to many services, which organically grew into a
number of new items for my toolbox. The end result combines common model
abstractions, low level calls and mocks to provide an easier and more uniform
cloud experience.

 

To interact with a
service you first need to initialize a connection:

 

    vcloud
= Fog::Terremark::Vcloud.new(

      :terremark_vcloud_username
=> INTENTIONALLY_LEFT_BLANK,

      :terremark_vcloud_password
=> INTENTIONALLY_LEFT_BLANK

    )

 

Ease of Use via Models

The common model
abstractions make it faster and easier to get started working with the basics
in several important ways. First by normalizing nouns across services, you can
just work with servers instead of needing to know that slices, instances, vApps
and servers have many shared operations. Next, they contain knowledge of what
information is required for different operations. This allows you to set up the
basic operations and depend on the models to let you know when you leave out an
important value. Finally, they provide helpers to make common operations
easier.

 

Some examples of the
model abstractions:

 

    #
list task models

    vcloud.tasks.all

 

    #
list task models (lazy_loaded)

    vcloud.tasks

 

    #
load task model by id

    task
= vcloud.tasks.get(task_id)

 

    #
use the wait_for and ready? helpers to wait for task completion

    task.wait_for
{ ready? }

 

Great Power and Responsibility via Low Level Calls

The low level calls also
provide important functionality. To start, they provide all the functionality
that the models provide a simpler interface to. They also provide an overflow
valve when the basic common functionality just won't cut it. In addition to
these important functions, they are the first part of a new cloud provider to
be implemented in fog, so they provide early access to functionality that the
models will later contain.

 

Some examples of the low
level calls:

 

    #
instantiate a vapp_template {12 => Ubuntu JeOS 9.10 (64-bit)}

    vcloud.instantiate_vapp_template('fog',
12)

 

    #
deploy a vapp_template

    vcloud.deploy_vapp(vapp_id)

 

 

Faster Development via Mocking

Last but not least, fog
provides mocks to make development faster and more pleasant. When fog is placed
in mock mode it accepts the same calls, but rather than sending out the actual
calls it just keeps track of the resulting state changes in memory. This allows
developers to try out provisioning code without having to wait for servers to
become available and without having to incur the cost of resources up front. These
are usually the last piece to arrive for a provider, so I have yet to add them
for vCloud, but this should provide a preview of goodness to come.

 

Mocks are enabled like
this:

 

    #
turn on mocking

    Fog.mock!

 

By Our Powers Combined

Putting these together
provides powerful tools for evaluating cloud providers and implementing
provisioning automation. vCloud is the newest provider in fog and currently has
many low level calls, a few models and no mocks. You can help implement more
support and follow its progress on github here or you can see a more
complete example of fog with vCloud here.

 

geemus (aka Wesley Beary) is an avid Rubyist and Open
Source enthusiast. When not spending his spare time working on enumerable
overly ambitious open source projects, he spends his days developing the Engine Yard Cloud.

 

 

SOA and Cloud Computing: Are They The Same?

By Steve Jin, VMware engineer

 

This entry was reposted from “DoubleCloud”
a blog for architects and developers on
virtualization and cloud computing.

 

SOA has been a buzzword in
enterprise computing for a long time now. Cloud computing is a relatively new
buzzword but it’s gaining momentum as SOA seems to be falling from favor.
Interestingly, with these two terms many people get confused: is cloud
computing
just a new name
for SOA?
And can they play well together?

 

Definitions

Before drawing a conclusion,
let’s take a look at the definitions from wikipedia.org:

 

Service
Oriented Architecture (SOA): is a flexible set of design principles used during
the phases of systems development and integration. A deployed SOA-based
architecture will provide a loosely-integrated suite of services that can be
used within multiple business domains.

 

Cloud
Computing: is Internet-based computing, whereby shared resources, software and
information are provided to computers and other devices on-demand, like a
public utility.

 

As we’ve seen in various
articles and books, SOA is mainly for enterprise while the cloud computing
definition suggests it’s for Internet-based services.

 

What’s Common?

In a previous blog, I mentioned
that
cloud as architecture pattern (CAA), which can be leveraged in enterprises as well. If we compare SOA and CAA, they look somewhat similar and share some core principles.

 

First, both emphasize the
service concept.
By definition, a service is performance of work by one for
another. Both CAA and SOA delegate work to other parts of the system, either by
the service provider or other business components. With that delegation, people
can use the services without worrying about the implementation details and
scalability. Most important, services in CAA and SOA can be shared by multiple
applications and users, thus 
optimizing resource utilization.

 

Second, both promote loose
coupling.
Each architecture demands minimum dependencies among
different parts of the system. As a result, any single change on one part of
the system has limited impact on the overall system.

 

What’s Different?

SOA and CAA differ significantly:

 

* Horizontal versus vertical services.

The services in SOA mainly focus on business. Each service may represent
one aspect of the business. Combined together, these services consist of a
business application or solution. In this sense, the services are horizontal.

 

The services in CAA are mainly layered according to typical software
stacks. The lower services support the upper services to deliver applications.
Therefore, I call them vertical services.

 

* Application vs. infrastructure

 

SOA is for application
architecture.
The dividing of different
components
is based on their roles in the SOA applications. More often
than not, you start with a business problem and then abstract out the services.
These services can be re-used by other applications in the future.

 

Cloud Computing is for IT
delivery. The dividing of different services is based on their roles
in a software
stack that is mostly well
defined. You don’t need a problem before defining the cloud services. The
services can be easily re-used by all applications.

 

Conclusion

SOA and cloud computing share many common principles, but also differ
significantly in their role in IT architecture. SOA is mainly an application
architecture with horizontal services; while cloud computing is an IT
architecture with vertical services.

 

Given the differences, SOA and cloud computing complement each other
very well. I will talk about SOA over
Cloud: A Perfect Match for Enterprise Computing
in a future blog. Stay
tuned.

 

VMware engineer
Steve Jin is author of 
VMware VI & vSphere SDK (Prentice Hall), founder of open source VI Java API, and is the chief blogger at DoubleCloud.org

Amazon AWS SDK for Java: It’s Not Quite There Yet

By Steve Jin, VMware engineer

Amazon released its SDK for Java last month. It’s a
complete set of APIs that cover almost all the services from EC2, S3, SQS, RDS
to SimpleDB. Overall Amazon has done a consistent job, but the SDK is NOT really
object oriented. And I think that is critical for higher development
productivity. 

Structure of the SDK

Before explaining why it’s not quite there, let’s take a
look at the structure of the API itself. The SDK includes many packages. For
each service, you find two or three packages: basic, model, and possibly util
package.

In each of the basic packages you can find the two
interfaces, and two implementation classes. Let’s pick EC2 as a sample here.
The AmazonEC2 and AmazonEC2Async are two interfaces and implemented by
AmazonEC2Client and AmazonEC2AsyncClient respectively. More methods are defined
in the synchronous than the asynchronous versions with the majority of them
overlapping with similar method names.

1

AllocateAddressResult allocateAddress()

2

AllocateAddressResult
allocateAddress(AllocateAddressRequest allocateAddressRequest)

 

3

Future<AllocateAddressResult>
allocateAddressAsync(AllocateAddressRequest allocateAddressRequest)

The first two versions wait and get you results upon return.
The third version doesn’t wait and gets the result later.

Overall Amazon has done a consistent job all over the
different types of services. The consistency is always a great attribute for
good API design.

What Should Have Been Done Differently?

If you have done Web Service programming before, you know
the SDK is about the same level as the stub generated by a Web Service toolkit
in terms of abstraction. With that, you have a fat “client” taking care of
everything, including all the possible operations. This is an easy and
straight-forward way for the SDK developers to come up with a SDK for the Web
Services. But it’s not so easy or straight-forward for the actual SDK users.

As most object-oriented (OO) programmers know, it’s a bad
idea to have everything in one fat class. If you have everything in one class,
then it’s not really OO programming. That is actually a typical mistake many
programmers make while transitioning from a procedural language like C to an
object-oriented language like C++.

So what about the classes in the model packages? These
classes are really data objects, whose methods are limited to getters and
setters. In other words, you can’t do anything meaningful with these data
objects. These data objects serve as the parameter and return types for the
methods in the two classes: one for synchronous and the other for asynchronous
calls. Some of them are actually over-used.

Let’s take a concrete example. The following is a method
from AmazonEC2Client class:

1

public void deleteSubnet(DeleteSubnetRequest deleteSubnetRequest)
throws
AmazonServiceException, AmazonClientException

The DeleteSubnetRequest is a data object which holds really
one string as a subnet ID. It would be better to have the signature as:

1

void deleteSubnet(String
subnetID); 

This is just a minor issue anyway. The bigger issue is that
the SDK doesn’t have real Subnet class to which you can add methods like
create(), delete(). If yes, you will find the application code can be as
straight-forward and easy as this:

1

Subnet sn =…

2

sn.delete();

Lack of Clear Object Model

The root problem with Amazon’s approach is the lack of a
clear object model, which I believe is the core of every successful OO API
design.

Nowadays, Web Services and REST are pretty popular. By
nature, they are all procedural. Focusing on these can easily shift your
attention from the core design to the design of the requests/responses schema,
URLs etc. Then you can easily lose the vision of the forest for the individual
trees.

The lack of an object model is not a big deal when the API
scope is small. When it gets bigger, we will see clear usability issues for the
API users.

VMware vSphere API has a great object model as described in
its API reference, but gets lost in the Web Services SDK due to the inherent limitation
of Web Services. Because the vSphere API is a lot more complicated than Amazon’s
SDK, users find it difficult to use. If you’re curious to learn more about how
VMware sets out to solve these same problems as Amazon’s latest AWS SDK, first
check out the open source vSphere Java API.
It re-creates the object model and for first-timers it can be much easier to
use than the vSphere Web Services.

VMware engineer Steve Jin is author of VMware
VI & vSphere SDK (Prentice Hall)
, founder of open source VI
Java API
, and is the chief blogger at DoubleCloud.org

Cloud Expo New York Tweetup April 20 with vCloud and Terremark

You’re invited! Come join VMware vCloud and our
partner Terremark at the infamous Irish
Rogue
pub and restaurant in New York City on April 20 from 6:30 – 8pm for
our first-ever vCloud Tweetup.

 

Register here
to reserve your seat by the fireplace!

 

Located at 356 W 44th Street in the Hells Kitchen
neighborhood, what better place for cloud heads to gather? Here you can
casually meet and greet some of the founding members of the vCloud ecosystem,
including:

*  William
Shelton, Director of Cloud Computing and Virtual Appliances at VMware

*  Jason
Lochhead, CTO of Hosting Services at Terremark Worldwide

*  Simon
West, Chief Marketing Officer at Terremark Worldwide

*  Adrian
Cole, founder of the open source jclouds framework and the cloudhackers
collective (see his video
talking about the benefits of using Cargo to automate app deployment to vCloud
Express)

 

Be sure to ask about how vCloud sets apart from the
rest, and what's new on the horizon…. and be sure to thank VMware and
Terremark for the snacks and drinks!

 

To keep
current on all vCloud Tweetups and other vCloud activities, follow us on
Twitter @vcloud or become a fan of our vCloud
Facebook
page.

Debut Cloudhackers Cloud Demo Hack Day

jclouds project founder Adrian Cole (see his video
talking about the benefits of using Cargo to automate app deployment to vCloud
Express) updates us on recent cloud discussions that we want to share with our
vCloud blog fans.

 

This week cloudhackers from
around the world met up in person and online for our first ever hack day. 
The epicenter was in San Francisco where the founding cloudhacker-SF meetup group
crashed CloudKick's headquarters for
the day.  cloudhackers-MTL
congregated at Station-C, and all our
distributed friends met up on freenode irc #cloudhackers.  

 

By 1pm, CloudKick's desks,
futons, and beanbags were full of hackers, attracted in no small part by an
extremely heavy order of 19 burritos, snacks and drinks, all compliments of VMware
vCloud Express
.  At the same time, our irc channel was full of
commotion.  It was a combination of newcomers, and lots of cool stuff from
cloudseed and Adam Jacob, CTO from Opscode.  So much chatter, that I think
we overran geemus' irc client ;)

 

Our goal was to add
content to clouddemo on github so that users could figure out what tools they
want without already knowing their names.  For example, some people know
they need to use chef and would prefer to use ruby.  Accordingly, we
decided to name cloudseed's
demo
“cloudseed-bootstrap-chef-ruby.md.” Pretty easy, right?

 

The afternoon continued
with several of us hacking improvements to our tools or explaining our tools to
others. Wesley Beary completed Terremark vCloud support in his ruby library
called Fog, which was amazing
progress in just a couple hours.  The dotcloud guys teamed up with
libcloud for an interesting discussion on cloud images continued on the
libcloud mailing list later that night.

 

One of the coolest things
we were able to accomplish was to decide on a logo!  Emie Lee from design
shop byemie.com
donated a day of creative services to our group and created a few options for our
logo.  After some local to San Francisco deliberation and vetting via irc,
we decided on the logos below (listed also on the
vCloud
Facebook
page)
: one is for Twitter and the
second for general use.  We love it and I hope you like it too!

 

It was really fun to have
everyone online, and great that Hugo Duncan
organized the Montreal
cloudhackers
to join us from Canada.  Feedback has been really
positive, so be sure we'll do it again.

 

Let
me know if you want to get involved with cloudhackers. Comment below or share
your thoughts with me on Twitter at @jclouds or at @vcloud. We’ll also be
talking about cloudhackers (with some photos) on the vCloud
Facebook
page. Become a fan!

Cloudhacker Twitter logo

CloudHackers_logo_twitter2
Cloudhacker general use logo

CloudHackers_logo_for_meetup

World’s First Redis Meetup

 

jclouds
project founder Adrian Cole (see his video talking about the benefits
of using Cargo to automate app deployment to vCloud Express) updates us on
recent cloud discussions that we want to share with our vCloud blog fans.

 

First Redis
Meetup

Engine Yard recently held a meetup on Redis,
the first in the world I hear. About 50 developers attended. It was a very very
very geeky crowd.  Very C, Ruby and Python centric, too, with some Clojure
fans as well. Java was not mentioned in discussions or asides.  

 

Developers are very happy with Redis, which
is essentially an atomic data structure server. They love the ease of creating
tools on its API, which was lamented as a problem in and of itself, since
everyone does it differently.  This “differently” problem surfaced in the
naivety of Redis, which doesn't handle scaling well.  For example, there
is only one master of data, so people implement sharding in the clients and
make up their own code (consistent hash) to pass requests to the right server.
 Not only is this difficult to get right, but since they are doing it
differently, you cannot use the perl Redis client on a Redis server being
accessed by the ruby one, since they don't use the same sharding approach. This
part feels very immature, but very consistent with the low clustering aptitude
present in other things like memcached.  

 

Redis is
Fast to Start

However limiting, it didn't seem
to be a big deal for a few reasons. One is that most developers cannot or do
not have the patience to setup traditional data grids.  Redis apparently
takes under two minutes to get running.  The other is that developers are
using Redis for extremely focused bits and only using one language.  For
example, Engine Yard is using Redis to deal with the polling problems of cloud
APIs like EC2.  Essentially, their pollers hit the Redis server, and update
only if a state change happened (like running to terminated).  Their other
code only pays attention to Redis, and therefore doesn't spam on multiple
pollers saying the same thing is terminated.  This is a very focused use
and all their code is ruby anyway, so it isn't a big deal. Moreover the data
size is measured in kilobytes so sharding is not useful.  Other uses were
to deal with deduping redundantly delivered messages.  This was due to
rabbit mq not having a good HA story outside erlang itself.  
Ezra Zygmuntowicz, a co-founder at Engine Yard, basically sets up redundant exchanges and has some
wrapper code to use Redis as a distributed lock.  Craigslist uses Redis as
a queue for spam filtering. There were a couple mentions of removing rabbitmq
with Redis in the future, as provoked by the audience.  The audience
absolutely loved Redis founder Salvatore Sanfilippo for being very responsive
24 hours a day and keeping the API easy to use and install.

 

Let me know if you’ve played with Redis yet. What do you
think of it? Comment below or share your thoughts with me on Twitter at
@jclouds or vCloud at @vcloud. We’ll also be talking about Redis on the vCloud Facebook page. Become a
fan!


Introduction to LabCloud – Part 1 of a Multi-Part Series

By Curtis Pope – vCloud Architect

 

Welcome to the first part of a multiple part
series on VMware LabCloud.  Many
VMworld 2009 attendees had the opportunity to be the first to experience the
"LabCloud" self paced lab environment developed by the VMware Cloud
Architect Team.  In fact, more than
4,500 individual labs with over 27,000 VMs were deployed over the 3 1/2 day
period.  In this blog entry, I will
give you a little more information on LabCloud and will follow it up over the
next couple of months with more detailed discussions of its architecture and
design.

 

Portal

 

First off, lets talk about the "Cloud
Problem" that LabCloud was built to solve.  As with all things cloud, the Cloud Architect team wanted to
bring a more "Self Service" model to our labs environment at
VMworld.  In order to move as many
people through the labs with the best experience possible, the administrative
burden had to be removed from the Technical Operations staff and placed into
automation on the equipment servicing the customer.  We also wanted to give the customer the choice of which lab
they wanted take in real time to better leverage the environment and adjust to
the demand.  Last, we needed to
provide the automation and self service environment in an easy to use front end
portal.

 

As a starting point, VMware Lab Manager 4
running on top of vSphere 4 was chosen as our preferred platform.  Some could say that Lab Manager was the
first VMware product to enable cloud like functionality.  How is Lab Manager "Cloud
Like" you might ask?  Well
lets see:  organizations could
build out their own "Development Cloud" using Lab Manager, allowing
end users to serve themselves while IT maintains control of the resources.  Lab Manager hides the underlying
infrastructure from the end user. 
Lab Manager allows a certain level of segregation of users and their
resources from each other.  All of
those things are key tenants of cloud computing.

 

We chose Lab Manager because it delivered a
couple of key features that allowed us to achieve fast deployment, dedicated
isolated environments for each user and repeatability of the lab experience.

 

First off, linked clones.  Linked clones is the ability for
multiple copies of a virtual machine to map back to a single parent disk.  When you deploy a virtual machine based
off a linked clone, a "Delta Disk" is created that stores the changes
that are made for that specific VM. 
While cloning a multiple gigabyte virtual disk might take several
minutes, creating a linked clone takes several seconds.  The speed of duplicating virtual
machines with linked clones is key to allowing us to bring up user environments
(vPods) very quickly.

 

The next key feature is the network isolation
provided by Lab Manager through the firewall router VM (FRVM).  The FRVM is part of the Network Fencing
technology that allows you to deploy a set of virtual machines (vPod) in an
isolated network environment. 
Network fencing allows the virtual machines in the vPod to be
pre-configured to communicate with each other.  Services such as DNS, DHCP, Active Directory, etc can be
provided inside the vPod while at the same time allowing multiple copies of
those vPods to exist in the same physical environment without conflict.  This fenced off network environment
also allows us to "stamp out" as many copies of a lab environment as
we need without worrying about IP address or name conflicts in the environment.

 

Network2

 

Lastly, the Lab Manager SDK/API.  Lab Manager has two APIs available for
use.  The fully supported API has
most of the functionality needed to create and deploy virtual machines.  The "Internal API" which is
used heavily by our QA test groups provides additional features and access to
advanced functionality like configuring the fence networking environment.  Both APIs are provided via SOAP and are
therefor usable by most modern programming languages.  We chose python as our preferred language and bindings to
this SOAP API are fairly simple through a SOAP abstraction layer called
ZSI. 

 

On top of the infrastructure provided by Lab
Manger and vSphere, we added our "LabCloud Portal".  The LabCloud portal provides all of the
end user interaction to the environment to allow starting and stopping of labs,
presentation of the lab manuals and redirection through RDP into the dedicated
individual lab environment.  The
LabCloud portal was written in Python leveraging the Django Framework for rapid
development.  Django provides
database abstraction and an easy to use template system to allow very quick
development of web based portals.

 

The portal shown above combined with Lab
Manager allows the user to click a single button which in turn checks out and
deploys a configuration (vPod) within Lab Manager, attaches it to the physical
network, returns RDP information back to the user and presents the lab manual
for the lab walkthrough.  When the
user is finished with his/her lab, they click on another single button to
"End Lab".  Another
automation process contacts Lab Manager, cleans up the configuration and resets
the station back to an unused state for the next user.  All of these deploy and destroy
processes happen without administrative action and generally in less than 5 to
7 minutes.

 

I
hope this blog has given you a taste of what is possible with Lab Manager and
has peaked your interest in what VMware is doing at events with LabCloud.  Check back in a few weeks for the next
installment where I will dig a little deeper into the architecture of LabCloud
and talk about our Multiple Lab Manager deployment model for scalability.