Home > Blogs > VMware vApp Developer Blog

VMware Studio 2.1 is now available for download

VMware Studio 2.1 GA was announced on Tuesday, July 13, 2010 in conjunction with the vSphere 4.1 release. The 2.1 release extends support for various OS and also brings enhanced flexibility, predictability and security to the Virtual Appliance development and update process.

• We listened to our customers and unlike previous versions, VMware Studio 2.1 can now create virtual appliances from virtual machines that were not originally created with VMware Studio, based on a discovery phase. In addition, while building an appliance, EULA(s) can now be retrieved from an input file in additional to being embedded in the build profile.

• Studio now enables users to create their own build profile templates. The CLI offers the –newos option using which you could generate a build profile for any RPM-based or DEB-based Linux OS.

• We also took a stride towards securing the appliance deployment and update process. We added support for digitally signed OVF files so that VMware vSphere 4.1 can verify the signed certificate during OVF import. In addition, using signed manifests during the update process, you could ensure that patches are coming from a trusted source.

• We also added a feature around localization. You can translate your EULA(s) into multiple languages, and vSphere Client as well as the VM first boot script will display it in the language of the prevailing locale. Also, you can now enforce EULA acceptance during the appliance update process.

• On the performance and optimization side, with Studio 2.1, you could now run concurrent builds. After a configurable limit is reached, builds wait in a queue. In addition, VMware Studio 2.1 optionally analyzes the list of RPM and DEB packages to locate unused items and generate a small-footprint virtual machine. It can also reduce the footprint post-installation with a file removal list.

• With Studio 2.1, during the appliance provisioning process, the generated virtual machine always reboots before your application installs. Some applications expect a full installed working system before they themselves will install. Rebooting after OS installation provides the real environment in which to install the application.

• A new utility program “vamisupport” is shipped with the output appliances for retrieving useful data for troubleshooting VAMI related issues.

• The new Linux OSs support we added are SLES 11, CentOS 5.4, RHEL 5.4, and Ubuntu 8.04.3 and 8.04.4.

In addition to the new features mentioned above, we fixed some of the major issues our customers faced with Studio 2.0. The issues around packages installation order (as specified in the Application Package Repositories) not always being honored has been fixed. Disk capacity set in build profile is now properly displayed in the web interface. VAMI is now fully functioning in SELinux enabled systems. Finally, OVF Tool used for transporting VM during provisioning has been upgraded to address known issues.

Download VMware Studio 2.1 GA release from here and give it a spin. We look forward to your feedback.

How to run Zimbra on Fusion with vApprun

Updated: To work with the latest version of the Zimbra virtual appliance (Sep. 6). 

The Zimbra folks have been so nice to create a virtual appliance that contains the entire Zimbra Collaboration Suite. The Zimbra virtual  appliance is packaged up in the OVF format and is using several advanced vSphere features (such as OVF properties) to  simplify the entire installation. In fact, if you have vSphere 4, it is a  really simple process to get Zimbra up and running.

But what if you don't have access to vSphere 4? (or just want to try it out on your laptop while eating lunch?)

In this post, I will show how you can use the freely available tools, OVF Tool 2.0.1 and  vApprun 1.0 to deploy, configure, and run the Zimbra appliance on VMware Fusion or VMware Workstation. Basically, we are tricking Workstation/Fusion to think it has all the nice OVF capabilities from the full-fledged vSphere product.

So before we start the pre-requisites are:

  • An installed version of VMware Fusion 3 or VMware Wokstation 7
  • VMware OVF Tool 2.0.1 (download)   
  • VMware vApprun 1.0 (download)      

Make sure that both OVF Tool and the vApprun command is in your path.

Step 1: Create a vApprun workspace

The vApprun tool implements a complete OVF / vApp environment for Workstation and Fusion. To use vApprun, we need to setup a vApprun workspace:

>mkdir zimbra
>cd zimbra
>vapprun init

Step 2: Configure the IP Pool

Each vApprun workspace has an IP pool that stores the network settings and other information about your deployment environment. We set that up as follows:

>vapprun workspace "dns=10.20.60.1;10.20.60.2" 
                   "gateway=10.20.63.253" 
                   "netmask=255.255.254.0" 
                   "range=10.20.63.200#4"
IP Pool settings:
  dns = 10.20.60.1;10.20.60.2
  domainName = example.com
  gateway = 10.20.63.253
  hostPrefix = vapprun-
  httpProxy =
  netmask = 255.255.254.0
  range = 10.20.63.200#4
  searchPath =
Free IP addreses
  10.20.63.200, 10.20.63.201, 10.20.63.202, 10.20.63.203

Make sure to substitute the values shown for DNS, gateway, netmask, and range with values that work in your environment.

Note 1: The above command should be executed as a single line. It has only been broken into multipe lines, so it is easier to read. The quotes ("") are only needed on Windows.

Note 2: If you are lazy and are on a network with a DHCP server, you can skip this step! (see step 5)

Step 3: Download the Zimbra Virtual Appliance

To download the Zimbra Appliance, open a web browser and go to: http://www.zimbra.com/products/download_appliance.html

You are required to fill out a registration form in order to get access to the appliance. 

Simply fill out the form, submit it, and you will receive an email with download 

instructions. Once you get the email, click on the download link (under Download Information)

to go to the download page. 

Download

On the download page, copy the link listed 

as: vSphere 4.0+ (Open Virtualization Format 1.0) – Direct Import URL.

 

Step 4: Use OVF Tool to deploy to vApprun workspace

Now it is time to download and deploy the ZImbra Appliance into our vApprun workspace.


>ovftool "–name=zimbra" <<Paste URL from above here>> .
Opening OVF source: http://files2.zimbra.com/…/ovf/zimbra.ovf
Warning: No manifest file
Opening vapprun target: .
Info:    End User License Agreement
License:
     VMWARE END USER LICENSE AGREEMENT

     IMPORTANT-READ CAREFULLY: BY DOWNLOADING, INSTALLING, OR USING THE
     SOFTWARE, YOU (THE INDIVIDUAL OR LEGAL ENTITY) AGREE TO BE BOUND BY THE
     TERMS OF THIS END USER LICENSE AGREEMENT (EULA). IF YOU DO NOT AGREE
     TO THE TERMS OF THIS EULA, YOU MUST NOT DOWNLOAD, INSTALL, OR USE THE
     SOFTWARE, AND YOU MUST DELETE OR RETURN THE UNUSED SOFTWARE TO THE
     VENDOR FROM WHICH YOU ACQUIRED IT WITHIN THIRTY (30) DAYS AND REQUEST A
     REFUND OF THE LICENSE FEE, IF ANY, THAT YOU PAID FOR THE SOFTWARE.

     … abbreviated …

Accept end-user license agreement?
Write 'yes' or 'no' (write 'read' to reread the EULA): yes
Deploying to vapprun workspace at: c:\Temp\zimbra\zimbra
Disk Transfer Completed
Completed successfully

Make sure to replace the URL in the above command with the one you got in Step 3. 

Notice that we just use a dot (.) as the target location. We can do that since the current directory is the vapprun workspace directory. 

Let's check that we got a Zimbra VM in our vApprun inventory:

>vapprun list zimbra
Name.......: zimbra
Type.......: VM
VMX........: .../zimbra/vmx/vm.vmx
Tag........: Zimbra_Collaboration_Suite_Appliance
AppUrl.....: http://${vami.ip0.Zimbra_Collaboration_Suite_Appliance}/
Transport..: com.vmware.guestInfo
IP Policy..: fixed
Property Definitions:
Key            Type      Value           UserConfig
-------------------------------------------------------------------------------
Zimbra_CONFIG_PASS    string(6..20)               True
Zimbra_HOSTNAME      string(1..65535)              True
Zimbra_ADMIN_EMAIL    string(1..65535)              True
Zimbra_ADMIN_PASS     string(6..20)               True
Zimbra_DISABLE_FW     boolean     False           False
Zimbra_DISABLE_NETWORK_TAB boolean    False           False
Zimbra_SERVICES      string     ALL            False
Zimbra_LDAP_MASTER_IP   string     @@LDAP_MASTER_IP@@     False
Zimbra_LDAP_MASTER_URL  string     @@LDAP_MASTER_URL@@    False
Zimbra_LDAP_ROOT_PASSWORD string     @@LDAP_ROOT_PASSWORD@@   False
Zimbra_ZIMBRA_LDAP_PASSWORD string    @@ZIMBRA_LDAP_PASSWORD@@  False
Zimbra_VAMI_USER     string     @@ZIMBRA_VAMI_USER@@    False
Zimbra_VAMI_PASS     string     @@ZIMBRA_VAMI_PASS@@    False
Zimbra_MTA_AUTH_HOST   string     @@MTA_AUTH_HOST@@     False
Zimbra_HYPERIC_HQ_HOST  string     @@HYPERIC_HQ_HOST@@    False
Zimbra_HYPERIC_HQ_USER  string     @@HYPERIC_HQ_USER@@    False
Zimbra_HYPERIC_HQ_PASS  string     @@HYPERIC_HQ_PASS@@    False
vami.DNS0.Zimbra_Collaboration_Suite_Appliance expression   ${dns:Network}    False
vami.gateway0.Zimbra_Collaboration_Suite_Appliance expression ${gateway:Network}  False
vami.netmask0.Zimbra_Collaboration_Suite_Appliance expression ${netmask:Network}  False
vami.ip0.Zimbra_Collaboration_Suite_Appliance ip:Network              True
vm.vmname         string     Zimbra_Collaboration_Suite_Appliance   False
Property Settings:
Key            Value
------------------------------------------------
Zimbra_HOSTNAME      CHANGE-ME
vami.ip0.Zimbra_Collaboration_Suite_Appliance
Zimbra_ADMIN_PASS     CHANGE-ME
Zimbra_ADMIN_EMAIL    CHANGE-ME
Zimbra_CONFIG_PASS    CHANGE-ME
... abbreviated ...


We need to configure the 4 properties listed under Property Settings (with value:
CHANGE-ME). We also need to decide how we allocate an IP address to the appliance. We can  setup the IP property to be configured from vApprun's IP pool (transient), using DHCP, or enter a fixed address. All in all this amounts to the following command:

>vapprun set-property -transient zimbra
                     "Zimbra_ADMIN_PASS=secret"
                     "Zimbra_CONFIG_PASS=secret"
                     "Zimbra_ADMIN_EMAIL=test@vmware.com"
                     "Zimbra_HOSTNAME=zimbra-host"

If you want to use DHCP, replace the -transient option above with the –dhcp option in the above command. And a

gain, the above command should be typed in as a single line.So let's double check that we did everything right:

>vapprun list zimbra
… abbreviated …
IP Policy..: transient

Property Settings:
  Key                       Value
  ————————————————
  Zimbra_ADMIN_PASS         secret
  Zimbra_CONFIG_PASS        secret
  Zimbra_ADMIN_EMAIL        test@vmware.com
  Zimbra_HOSTNAME           zimbra-host
… abbreviated …

Step 5: Launch Zimbra 

The moment we have been waiting for… let's start it:

>vapprun start -gui zimbra
Starting zimbra
Access at: https://10.20.63.200/

The Zimbra appliance takes a few minutes to go through the first-time boot configuration. Once it is up and running, you can access the appliance at the URL above. To login use the admin email and password you configured in step 5 (test@vmware.com, secret).

Note: If you are using DHCP, then vApprun does not know at the time of power-on what IP address will be used, so the Access at: message will not be shown. Instead, run the vapprun list command to see what IP address has been allocated to the VM. This is reported back to vApprun through VMware Tools.

 


Zimbra
 

Once you have done the above steps a few times, it actually becomes quite simple. Finally, all of the above works equally well on Mac OS X, Windows, and Linux.

/Rene

OVF Tool 2.0.1 is now available for download!

We are happy to announce that OVF Tool 2.0.1 was released yesterday. It can be downloaded from here, and is available for Windows, Linux, and Mac OS X (new!). 

This is the first major release since the initial release of OVF Tool 1.0 and a minor upgrade to the OVF Tool 2.0 version that is bundled with Fusion 3.1 and Workstation 7.1. I have included an excerpt from the README below which describes the changes:


Changes since 2.0.0
——————-
. Fixed bug that caused OVF Tool not to work with the “Import from vSphere”
feature on the VMware Virtual Appliance Marketplace.
. Fixed launch issue on Linux/Mac if installed in a location where the path
includes spaces.
. Fixed issue with properties when importing to vApprun workspace.

Changes since 1.0.1
——————-
. Support for the OVF 1.1 specification.
. Support for OVF packages that includes ISO and floppy images.
. Support for vApprun as target and source.
  (see http://labs.vmware.com/flings/vapprun)
. Support for OVF Tool on the Mac OS X operating system (Intel-only).
. A lax mode (–lax) that enables a best-effort conversion of OVF
packages which do not fully conform to the OVF standard or include
virtual hardware that is not supported by the destination. For example,
VirtualBox (version 3.0 and 3.1) produced OVF packages can be converted in lax mode.
. Bug fix for handling vmxnet3 targets.

Please use the forum on the download page to post any comments or issues with the tool. We would like to hear to hear from you.

/Rene

The VMware Studio vApp Profile: Adding Existing OVFs

To create a vApp, VMware Studio uses a separate profile to specify the VMs to  include when constructing the  vApp, as well  as other various vApp-level parameters.


The  VMs  can  be specified  in  a  few  different  ways in  the  vApp profile. The relevant section in the XML looks something like this:


    <vadk:VMCollection>
        <vadk:VM
            vadk:ovfid="centos53"
            vadk:appurl="true"
            vadk:profile="centos53"
            vadk:location="http://[VADK.localIP]/build/centos53.1/exports/ovf/VM_OVF10.ovf"/>
        <vadk:VM
            vadk:ovfid="centos53 1"
            vadk:appurl="false"
            vadk:profile="centos53"
            vadk:location="http://[VADK.localIP]/build/centos53.1/exports/ovf/VM_OVF10.ovf"/>
    </vadk:VMCollection>

The vadk:ovfid attribute is a unique identifier for this VM. This will be used as a prefix to make any identical names in each VM unique. For example, the default Network label in a VM created by VMware Studio is called "Network 1".  In order for the VMs to be  able to have distinct networks, the   VM's   Network   labels  are   prefixed   with   the vadk:ovfid. Note that when vadk:ovfid is used to prefix filenames, any spaces in the vadk:ovfid will be replaced with underscores (_).


The vadk:appurl attribute is a boolean value  specifying that vCenter should use this VM's AppUrl value for the link in the Summary tab.


The  vadk:profile attribute  is  a pathname  to  the associated  build profile  for this  VM.  If  it  is solely  a filename  without an  xml extension (as it is here: centos53),  the filename is assumed to be in the     default    directory     where    profiles     are    created: /opt/vmware/var/lib/build/profiles/centos53.xml.    Note   that   this profile must  be configured  to produce an  OVF or  OVA as one  of its output formats (vadk:DistributionFormat).


The vadk:location attribute is where the location of an existing VM is specified. If vadk:location is empty,  then VMware Studio will use the value of  the vadk:profile attribute,  build the VM described  by this profile, and then include the resulting OVF or OVA in the vApp.


If the  vadk:location attribute is not empty,  it can contain one of several things:

  1. An absolute path (on VMware Studio) to a version 1.0 OVF or an OVA containing a version 1.0 OVF,
  2. A URL to a version 1.0 OVF or an OVA containing a version 1.0 OVF. In this URL, the keyword [VADK.localIP] may be used, and will be substituted with the actual IP address of the VMware Studio machine.


Note that the OVF or OVA specified does not need to have been built by VMware Studio. In particular, a  VM exported from vCenter can be used, as well as a third-party OVF or OVA. However, the use of a third-party OVF or  OVA is only supported on  a case by case  basis. VMware Studio cannot guarantee that the third-party OVF or OVA will combine properly into a vApp,  as the OVF specification is complex,  and it is possible that  a valid  OVF  may not  be  consumed by  the  VMware Studio  vApp building utility.


Let the VMware Studio team know of any difficulties you have with vApp construction, and we'll try to help.


vApprun, OVF 1.1, and ANSI/ISO Standards

In has been a while since the last update and a lot of exciting things has happens with regards to both OVFs and vApps in the meantime.

In the real code department:

I am very happy to annouce that  we just released our latest OVF-related fling on labs.vmware.comvApprun 1.0. This is a free, command-line tool that implements vApps and, OVF properties, and OVF environments for both Workstation and Fusion users. Thus, vApps are no longer just for vSphere 4. From the vApprun release notes:

A vApp is a container for a distributed, multi-VM software solution. It has power-on operations just as a virtual machine, and can be imported or exported as an OVF package. The OVF properties and OVF environment provides a flexible mechanism for parameterizing guest software inside a VM upon deployment. Using the OVF properties, it is possible to both configure OS level properties, such as IP settings, and application level parameters, such as IP addresses of external servers. The features of vApprun include:

  • vApps that contain multiple VMs and nested vApps.
  • For a vApp, the start/stop sequence of the child entities can be configured.
  • Start/stop/shutdown of vApps.
  • Create OVF properties and access the OVF environment from guest software.
  • Supports ISO and guestInfo OVF environment transports.
  • Supports fixed, transient, and DHCP IP allocation modes and IP pools.
  • Import OVF packages and run it with vApprun (using OVF Tool 2.0).
  • Export vApps and VMs created with vApprun to OVF packages (using OVF Tool 2.0).
  • Command-line, workspace-oriented user interaction.
  • Cross-platform – available for Windows/Mac/Linux.
  • VMware vSphere 4 compatible.

As a teaser, here is how to create a vApp with 2 VMs and a property:

>vapprun init
>vapprun create-vapp MyVApp
>vapprun create-vm VM1
>vapprun edit VM1 parent=MyVApp
>vapprun create-vm VM2
>vapprun edit VM2 parent=MyVApp
>vapprun def-property MyVApp key=ip type=ip:Network
>vapprun list MyVApp
Name.......: MyVApp
Type.......: vApp
IP Policy..: fixed
Children:
Name    Order StartWait WaitForTools StopWait
------------------------------------------------------------
VM2    30  30   True   30
VM1    30  30   True   30
Property Definitions:
Key      Type   Value      UserConfig
-------------------------------------------------------------------------------
ip      ip:Network         True
Property Settings:
Key      Value
------------------------------------------------
ip

You can download it here: vApprun 1.0 and start playing with vApps today. We will soon follow up with an more in-depth look at vApprun on this blog.

In the specification department:

The OVF  1.1 specification was released in January. This is a minor, fully backwards compatible update to the 1.0 specification. From the DMTF newsletter:

There are several new components that differentiate the OVF 1.1 specification from the original version, including:

  • Capability for file system-based images to increase flexibility at deployment time
  • A property attribute to hide password values at the user interface
  • Joliet extensions for ISO transport image
  • Clarification of how the OVF manifest contain entries for individual file chunks
  • Clarification of the referencing of manifest and certificate files from descriptor

In even more exciting news, the OVF 1.1 specification has been submitted to ICITS to become an ANSI standard, which is the first step to become an ISO standard. From the DMTF site:

DMTF has been busy extending and refining the Open Virtualization Format (OVF) specification. The organization recently completed version 1.1 of the spec and has submitted it for consideration as an ANSI and ISO standard.

This is an important milestone for DMTF. The specification will be put through a number of tests to ensure it meets a complex system of checks and balances of reviews before it is adopted as an American and International standard.

That is all the news for this time. Take care,

/Rene

Adding Linux Management Service to Virtual Appliance

VMware Studio 2.0 allows author to add their own in-guest services that will show up as another tab in the appliance web interface. If you want to add additional appliance configuration capabilities or want to consolidate appliance and application configuration into single web interface, this blog will show how to achieve the same easily. 

Lets create a simple in-guest service, that can run
arbitrary commands on the appliance. Since it is allowed to run arbitrary
commands we need to ensure that the service has authentication and can only be
invoked by a user with proper credentials.

Eclipse Plugin for Studio (see installation instructions at VMware Studio Eclipse Plugin
) provides an easy way to create and build Management Service. You can also
create the management service by hand by constructing the same directory
structure, and having a service.xml please see VMware Studio Developer Guide
for details of the format of the Management Service.

Create an new Project, and pick “VMware Studio Linux
Management Service Project” as shown below.

001

002

Enter details about your service.

003
 This will create a directory structure for the service as
shown below.

 004

 Generated service.xml has the details of the service, you
can modify this to fit your needs. If there are any external package
dependencies you can add them in service.xml.

 

<?xml version="1.0"
encoding="UTF-8"?>

<service>

  <name>myservice</name>

  <version>1.0.0.0</version>

  <description>My Service to
run a command
</description>

  <!– Dependencies

   <requires>

      <packages>vmware-studio-vami-servicebase</packages>

     
<services>network,system</services>

   </requires>

   –>

</service>

View.xml in the view folder points to what will be shown to the user in the
management tab of the VM, by default it points to an index.html page you can
change this to any other content. You will put all your html related content
for the service in the view folder, they will be placed in the
/opt/vmware/share/httpd/service/myservice folder on the Appliance.


<service>

      <title>My Service</title>

      <content type="url"
url="index.html" />

</service>


For our service we would want to invoke some command on the
Appliance and show the result to the user. Lets add the necessary html to take
input from the user in the index.html page.

<script language="JavaScript">

// Obtain the
vami object, please read documentation for various functions that can be
executed on vami

var vami =
parent.vami;

// Callback to
take the command result and show to the user

function
commandOutput(commandresult) {

      document.getElementById('output').value =
commandresult;

}

function runCommand() {

      var commandToRun =
document.getElementById(
'command').value;

      //cgi files are placed are served from
/service/myservice/cgi/

      var cgiToInvoke = '/service/myservice/cgi/backend.py?command=' + commandToRun;

      //Invoke vami.io to perform a httpGet
request with basic authentication and callback with the result to
commandOutput 

      vami.io.httpGet(true, true, 'Running the
command..'
, cgiToInvoke, commandOutput);

}

</script>

<h3>Sample CGI</h3>

<p>Run any command
on the appliance and get the results.
</p>

Enter the command

<input type="text"
name="command" id="command"
size="50" value="pstree">

<button type="submit"
onclick="runCommand()">Run</button>

<p><textarea class="output"
id="output" readonly="true">Command output >>></textarea>

</p>

</body>


vami.io.httpGet is a VAMI Web API that does AJAX call to the back end
and callbacks a function when it is done. Refer to VMware Studio Developer Guide
for detailed list of VAMI Web API.

 

We need to take care of running the actual command on
the Appliance, you may wish to write the backend in any backend technology you
may wish, you can even write it as a CIM Provider on sfcb.  

 

We will write our backend in python, lighttpd on the
appliance is already configured to work with python. If you would like to write
the backend in php, perl or ruby make sure you add the necessary lighttpd.conf
file for the service to configure lighttpd to work with those technologies.

 

Our backend code in python will simply invoke the command
return the result.

 

#

# Back end code
that will be executed to return data to the user

def backend_code():

   #return http header, the extra
empty line is required

   print 'Content-Type: text/html'

   print 'Status: 200'

   print 

   form = cgi.FieldStorage()

   #Obtain the form fields

   command = form['command'].value

   #Run the command as passed by the user
with current directory set to /root

   pobj = subprocess.Popen(command, bufsize=1, shell=True,

                       stdout=subprocess.PIPE,

                     stderr=subprocess.STDOUT,
cwd=
'/root')

  

   #Output of the command is sent back to
the user

   while True:

      line = pobj.stdout.readline()

      if not line:

         break

      print line.strip()

   sys.exit(0)

 

When we are invoking the backend we asked VAMI to do basic
http authentication, using vami.io.httpGet(true , we would need to add
the necessary checks in our backend code that we will only execute a given command
if user is authenticated. User name and sessionid is sent as part of the http
basic authorization, we can use this information to make a call on the SFCB
server to make sure that user is authorized.

 

def autentication_failure():

   print 'Content-Type:
text/html'

   print 'Status: 401'

   print 'WWW-Authenticate: Basic
realm="Authentication required"'
  

   print

   sys.exit(1)

 

if "HTTP_AUTHORIZATION" in os.environ:

   #of the form 'HTTP_AUTHORIZATION':
'Basic cxadsfas='

   http_auth = os.environ["HTTP_AUTHORIZATION"].split(" ")[1]

   user, sessionid =
base64.decodestring(http_auth).split(
":", 1)

   #Authenticate user using CIM server to
process username,password

   try:

      cliconn = pywbem.WBEMConnection('http://localhost:5488', (user,
sessionid))

      cliconn.EnumerateInstanceNames('VAMI_ComputerSystem')

   except Exception, e:

      autentication_failure()

   backend_code()

  

else:

   autentication_failure()

 

 

To Build the Service, right click
build.xml and select run as “Ant Build”, this will create a tar.gz.

 

005

 

You can now
include this service as part of your appliance, please refer to “Importing a
Management Service” section in VMware Studio Developer Guide and when you do, your final appliance web UI will be as shown below.

006 

 Download myservice.zip to get the full details of the code and the eclipse project.

Updates for VMs created with VMware Studio 2.0

Introduction to appliances and updates

VMware Studio builds VMs. As
a part of VM building, VMware Studio 2.0 embeds an in-guest agent also known as
VAMI (Virtual Appliance Management Infrastructure) for Linux VMs, this in-guest
agent allows users to configure system parameters, example: timezone, configure
networking parameters, example: specifying ip addresses, and optionally update
the VM.

A VM with update agent
installed/enabled is considered to be a virtual appliance, and the user of the
appliance can update the appliance with a single-click. The update model
provided by VMware Studio for appliances is a pseudo-firmware model, where a
single update will update all the components (application packages) in the
virtual appliance. The update includes both OS updates, and updates for the application,
this ensures that an update applied to the appliance doesn’t break the
application.

Appliance builder can publish
updates easily using Studio 2.0, updates are published to an update repository as a part of an appliance
build. An update consists of a set of packages that are in native package
formats (i.e. either rpms or debs based on the Linux flavor), as a result
updates can be tracked using existing package managers, and as an appliance
builder updates for non-application packages (OS updates) are easier to find.

Publishing Updates for your Appliance

Updates are published during build

During the creation of a
build profile to build a VM, the appliance author specifies the list of
application packages and OS packages that are to be installed for the
appliance.

When Studio builds the VM, it
can determine all the packages that were installed on the appliance, Studio
uses this information to create an update repository as a part of the build.
The update repository contains a manifest file which lists all the packages
that were installed and a pool of packages. This update repository can then be
used for testing updates and once tested it can be published externally, so
that the appliances in the field can get their updates.

Enabling updates in Studio UI

While creating the build
profile for your appliance you can add the update management service to enable
updates for your appliance, once this service is enabled the build profile
wizard will ask you to enter additional details.

Note: To publish to the same update repository the appliance version needs to be incremented.

a) Repository Access Methods

Repo_settings
The appliance will use either a CD-ROM or will connect to
the specified URL to obtain updates. Later
on, the appliance user can also point to a local (alternate) repository to
obtain updates, this can be configured by the appliance user using VAMI web
console.

b) Repository Server Settings

Repo_server

As a part of the build, Studio will
create an update repository at the specified location using the SCP information
entered. Repository server is usually the staging server (see next section for
more information) for the appliance updates.

c) Repository Export Settings

Repo_export

If you’d like to save a copy of the
update repository in either zip format, or create a CD-ROM based repository that
can be used to update the appliance it can be specified here.

Please refer to Studio
Developer's Guide documentation for more information.

Staging Server

A staging server is usually a
location within the organization. This location is used by Studio to publish
the updates during appliance build. This repository can also be used to test
the update before releasing the update.

Advanced Option – Update scripts

Studio provides ability for
the author of the appliance to specify scripts that can be run before or after
installation of packages (i.e. rpms or debs). A good example of such a script
would be: deletion of a package which already is present in the VM and would
cause conflict with a package in the update.

To edit these scripts, open
the build profile using an XML editor (located at
/opt/vmware/var/lib/build/profiles directory in the Studio VM), and change the
default content of <vadk:PreInstallShellScript> and/or  <vadk:PostInstallShellScript> under
section “vadk:UpdateSection_Type”.

Note: Scripts added will
have to be XML encoded to ensure that it is valid XML.

Updating your Appliance

The appliance can be updated
by the appliance user using either the VAMI Web Console, CLI on the appliance,
or using vSphere Client/vCenter.

Web Console

Using the VAMI web console,
the appliance user can check for updates, update the appliance, schedule
automatic updates, and specify alternate methods of obtaining updates.

If updates are being received
from an external repository then the user simply clicks on the Check button to
query if any updates are present, and if there are any updates present user
clicks on the Install button to update the appliance.

If an update is received as a
CD-ROM ISO file, then the user can attach the ISO file to the CD-ROM device of
the appliance, and run check/install operations as a user would for an external
repository.

CLI

Studio in addition provides
vamicli, a command line tool on the appliance, using which the appliance can be
updated.

Please refer to Studio User’s
Guide
for more information.

Testing Updates

It is recommended that the
author of the appliance, test updates before releasing the update to the
appliance users. In order to test the appliance which is built with an external
repository URL, the author can deploy the appliance inside the organization,
and change the repository URL to the staging server URL or use a CD-ROM based
update using the VAMI web console.

Integration with vCenter for centralized update
management

VMware vCenter Update Manager
a vCenter plugin allows users of appliances (built with VMware Studio) to
update their appliances. The remediation process during which the appliance is
update can either be started manually or as a scheduled task. With Update
Manager multiple appliances can be updated at the same time, when an appliance
boots up VMware Update Manager automatically discovers that the VM has VMware
Studio’s update agent in the VM and allows users to scan and remediate
appliances. 

Users can also define
baselines. Baselines are rules that the administrator of an appliance can
define using Update manager, an example would be: an appliance must always be
at the latest version. Using baselines the administrator can check appliances
for compliance.

More information on how to
use Update Manager can be found in the Studio User’s Guide.

Have fun publishing updates for your appliances!

 


Introduction to the Eclipse plugin in VMware Studio 2.0

There are many steps in developing, building and testing VMs and vApps using VMware Studio.  To make it easier Studio 2.0 adds a new tool to help, the VMware Studio Eclipse plugin. The plugin makes life simpler to many users by adding simple ways to package applications and build the application along with the VM or vApp, all in a single click.  To demonstrate the simplicity, in this blog entry, I will walk through an example of how to develop a simple web-based VM or vApp.

To follow along you will already need to have VMware Studio running and will need to set up an Eclipse development environment.

To get started I downloaded the Eclipse IDE distributed by Aptanu at http://www.aptana.org/ and installed the optional Aptana Aflax feature by going to Help->Install Aptana Features then selecting Ajax Libraries->Aflax and then clicking install.  You can also select the editing tools that you want to use for your app.

I then installed the VMware Studio Eclipse Plugin by following the guide at http://www.vmware.com/support/developer/studio/studio20/eclipse_plugin.html.

Now the normal application development tasks take place, I will just use a sample project that is provided by Aptana as my application.  If the Samples view is not open in your environment, then you can open it by selecting Window->Show View->Other…->Aptana Views->Samples then clicking OK.  Once the view is open, I will find the sample named spirograph, right click on it and select Import Sample as Project...

Once the project is opened I used the Project Explorer view to create a new directory named html and moved all the web files into it.  I then created a VMware Studio Package for Linux (.vsp) in the project and called it mypackage.  To created the vsp file, right-click on the project name and select New->Other…->VMware Studio->VMware Studio Package for Linux .

You can use the Eclipse tools to further test and develop this web app.  Open the index.html file and modify the source or preview the changes.


Once you are satisfied with the app, it is time to package it.  Open the mypackage.vsp file and fill out the package details.  This application requires a httpd server with php on the VM which is provided by the operating system, so add them to the dependencies line.

Now set up the file mapping.  The file tree on the left represents how the files will be laid out in the VM after it is built.  The source location is where the files are on your local machine.  For this application, we will just set the top level html directory, Eclipse will recursively go through all the files and automatically add them at build time.



This application needs the web server to automatically start when the VM is booted.  To make this happen, we edit the post-install script for vsp.



We are almost done!  Now we just need to set up a VM build profile on VMware Studio.  For this blog I chose to use CentOS and keep the default configuration for everything except setting the root password and editing the settings for the provisioning engine.

You can create the profile directly in Eclipse by clicking on VMware Studio Web Console in the VMware Studio Explorer view.  Create the profile then save and close it.

Then when you are ready, click on Build Package… in the vsp editor.  The first time you may be prompted to create a package repository, just accept the default by clicking Yes then OK.  Select the VM profile to add this package to, and enable Automatically start VM build.  Then click on Finish.

A new Eclipse job is started, which monitors the build, and will notify you when it is finished.  You can monitor the status of the build by opening the Progress view.


Once the build is done you can deploy the VM to test it.




That is all that is needed.  Easy!  Your VM is now ready to be shipped.

The Eclipse plugin provides several other feature, like looking at the build logs, and creating default service packages to extend the VAMI framework.

Please leave comments and let us know how this went for you!


Virtual Hardware in OVF – Part 1

In the blog post Inside the OVF Package, we talked about the structure of an OVF package, including the different sections (XML elements). One of the more important sections is the one describing the virtual hardware configuration for a VM. The operating system and applications installed in a virtual machine (commonly referred to as just the guest) are highly depended on the virtual hardware that is provided by the deployed virtualization platform. For example, if the hypervisor is exposing the virtual disks through an IDE controller to a guest that is expecting to use a SCSI controller, it is likely that the software will not boot or even worse it might even misbehave. A key objective of the OVF format is to ensure robust and well-behaved deployments. This is achieved by the OVF package author describe the requirements to the virtual hardware in the OVF descriptor, and the OVF deployment can check whether it can fullfill those requirements at deployment time.

In this first blog entry, we will introduce the hardware description model along with a simple examples. This is probably all that you need to know for most daily use. We will follow up with another post with some more examples, and finally with a post with some more advanced examples using deployment-time configurations and multiple hardware profiles.

Ok, lets, get started.

The OVF specification version 1.0 defines a VirtualHardwareSection. This is based on the ResourceAllocationSettingData (RASD) elements from the DMTF CIM standard. For users not familiar with the CIM model this may appear slightly complex at first, but there are many advantages to leveraging an already existing standard, and as we will see once you get used to it, it is pretty straight forward.

The 'System' Element:

We start with the System element found as <System> in the OVF descriptor. The System element is optional but is generally used to specify the “virtual hardware family type”. For VMware products we use the form ‘vmx-X’ where X is VMware's virtual hardware version number. The import process will convert the virtual hardware to use this version number. Several hardware versions may be defined by using space as a delimiter e.g. “vmx-04 vmx-07”. The list of hardware versions are unordered, meaning that the import process is free to pick a hardware version satisfying the remaining of the hardware requirements. VMware products will always pick the recommended hardware version of the host if multiple acceptable versions exists.

The remaining fields are not necessary for import but required by the CIM standard. A system element could look like:

<System>
        <vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
        <vssd:InstanceID>0</vssd:InstanceID>
        <vssd:VirtualSystemIdentifier>MyVm</vssd:VirtualSystemIdentifier>
        <vssd:VirtualSystemType>vmx-07 vmx-4 myHardwareVersion someOtherHardwareVersion</vssd:VirtualSystemType>
</System>

The following table show which hardware versions are supported on various VMware platforms.

Hardware Family Supported by (and later versions of the same products)
vmx-04 WS 5.0, Fusion 1.1, Server 1.0, ESX 3.0
vmx-06 WS 6.0, Fusion 1.1, Server 1.0
vmx-07 WS 6.5, Fusion 2.0, Server 2.0, ESX 4.0

The 'item' Elements

Virtual hardware is modeled as a set of devices, such as ethernet card or disk controllers, memory. Each of those devices are described by an element. The  DMTF RASD specifies a set of fields that can be set. We only a subset of those fields in an OVF descriptor. The fields in a RASD must be ordered alphabetically. The following table lists the RASD fields that are used:

ElementName Required field that contains a display-friendly message about the content of the RASD
Description Human-readable description
InstanceID Required field that contains unique ID within this <VirtualHardwareSection>
ResourceType Required field that indicates the kind of resource
ResourceSubType A vendor-specific identifier for specific devices
VirtualQuantity Specifies the amount (used for memory and CPU)
AllocationUnits Specifies the unit of resource (used for memory and CPU)
Reservation Specifies resource allocation policy (CPU and memory)
Limit Specifies resource allocation policy (CPU and memory)
Weight Specifies resource allocation policy (CPU and memory)
Address Typically used as the unit number of a controller
Parent Instance ID of parent controller (for devices on a controller)
AddressOnParent Used to specify the order for devices on a controller
AutomaticAllocation Used to specify whether a device should be connected on power-on (e.g., for a CDROM)
Connection Reference to a network for an ethernet adaptor
HostResource Reference to a virtual disk for a disk drive

The minimum Item element looks like this:

<Item>
<rasd:ElementName>SomeName</rasd:ElementName>
<rasd:InstanceID>1</rasd:InstanceID>
<rasd:ResourceType>0</rasd:ResourceType>
</Item>

ElementName is a general description of the Item element. InstanceID is a unique identifier for the Item element and is used to refer to other hardware elements. The ResourceType describes the type of hardware. The resource types used in OVF are described in CIM_ResourceAllocationSettingData.mof, which describes the mapping from the ResourceType number to hardware element type. For instance, ResourceType=2 maps to “Processor” (CPU) and ResourceType=10 maps to “Ethernet Adapter”. See the following table for the typical used ResourceTypes: 

Kind ResourceType
Other 0
Processor 3
Memory 4
IDE Controller 5
SCSI Controller 6
Ethernet Adapter 10
Floppy Drive 14
CD/DVD Drive 15/16
Disk Drive 17
USB Controller 23

For some ResourceTypes it is necessary to define a more specific subtype. ResourceSubType is vendor specific, meaning that VMware supports one set of subtypes while other vendors may support other types. ResourceSubType with ResourceType=0 (Other) is also used to define Virtual Hardware that is not described in the CIM schema, for instance a soundcard. The complete list of ResourceTypes with supported ResourceSubTypes for VMware products is shown below:

Kind ResourceType ResourceSubType Other Fields
Other 0 vmware.soundcard.sb16, vmware.soundcard.ensoniq1371, vmware.pcicontroller,vmware.ps2controller, vmware.siocontroller, vmware.keyboard, vmware.pointingdevice Depend on hardware type
Processor 3 AllocationUnit, VirtualQuantity, Reservation, Limit, Weight
Memory 4 AllocationUnit, VirtualQuantity, Reservation, Limit, Weight
IDE Controller 5 Address
SCSI Controller 6 lsilogic, buslogic, lsilogicsas, virtualscsi Address
Ethernet Adapter 10 E1000, PCNet32, VmxNet, VmxNet2, VmxNet3 AddressOnParent, AutomaticAllocation, Connection
Floppy Drive 14 AddressOnParent, AutomaticAllocation
CD/DVD Drive 15 Parent, AddressOnParent, AutomaticAllocation
Disk Drive 17 Parent, AddressOnParent, HostResource
USB Controller 23

Example

We will now look at a sample OVF descriptor that describe a small VM, that are using a single CPU, 512 MB of memory and the use of a disk on a SCSI controller. In this example we will look at the virtual hardware description and will disregard the the other metadata information present in the OVF descriptor. We will look at each of the Item elements in the virtual hardware section and explain what they do and how they reference each other.

The System element specifies that this hardware description is designed for vmx-07 or vmx-04: 

<Info>Virtual hardware requirements</Info>
<System>
        <vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
        <vssd:InstanceID>0</vssd:InstanceID>
        <vssd:VirtualSystemIdentifier>My Small VM</vssd:VirtualSystemIdentifier>
        <vssd:VirtualSystemType>vmx-07 vmx-04</vssd:VirtualSystemType>
</System>

If we look at the two first Item elements we first see a definition of how many CPUs we want the VM to have. The amount is in the VirtualQuantity element and for this VM we want 1 CPU. In AllocationUnits we are using the Programmatic Units as defined in http://www.dmtf.org/standards/published_documents/DSP0004_2.5.0.pdf. The next Item element we are defining the amount of memory.

<Item>
        <rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
        <rasd:Description>Number of Virtual CPUs</rasd:Description>
        <rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
        <rasd:InstanceID>1</rasd:InstanceID>
        <rasd:ResourceType>3</rasd:ResourceType>
        <rasd:VirtualQuantity>1</rasd:VirtualQuantity>
</Item>
<Item>
        <rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
        <rasd:Description>Memory Size</rasd:Description>
        <rasd:ElementName>512MB of memory</rasd:ElementName>
        <rasd:InstanceID>2</rasd:InstanceID>
        <rasd:ResourceType>4</rasd:ResourceType>
        <rasd:VirtualQuantity>512</rasd:VirtualQuantity>
</Item>

We now define a SCSI controller of subtype ‘lsilogic’ and with an InstanceID=3. We will later use the InstanceID to bind a disk to this controller. As can be seen we have not defined a PCI controller to bind the SCSI controller to. For VMware products PCI controller is a default device so there it is optional to add it as an Item element. The next two Item elements are defining IDE controllers.

<Item>
        <rasd:Address>0</rasd:Address>
        <rasd:Description>SCSI Controller</rasd:Description>
        <rasd:ElementName>SCSI Controller 0</rasd:ElementName>
        <rasd:InstanceID>3</rasd:InstanceID>
        <rasd:ResourceSubType>lsilogic</rasd:ResourceSubType>
        <rasd:ResourceType>6</rasd:ResourceType>
</Item>
<Item>
        <rasd:Address>1</rasd:Address>
        <rasd:Description>IDE Controller</rasd:Description>
        <rasd:ElementName>IDE 1</rasd:ElementName>
        <rasd:InstanceID>4</rasd:InstanceID>
        <rasd:ResourceType>5</rasd:ResourceType>
</Item>
<Item>
        <rasd:Address>0</rasd:Address>
        <rasd:Description>IDE Controller</rasd:Description>
        <rasd:ElementName>IDE 0</rasd:ElementName>
        <rasd:InstanceID>5</rasd:InstanceID>
        <rasd:ResourceType>5</rasd:ResourceType>
</Item>

We now want to add a CDROM drive to one of the IDE controllers. If we look at the Parent element we can see we want to bind it to a device with InstanceID=5. If we look for an Item element with InstanceID=5 we can see that this is IDE 0. We use AddressOnParent to tell where on the IDE controller we want to place the CDROM. If no AddressOnParent element is specified devices are attached to their “parent” in the order they are found in the OVF. Here we see that the AddressOnParent=0, so we set it to device 0 on the IDE controller. On the Item element we are defining a ovf:required=”false” this tells the import process that if we are unable to add this hardware device the import process is allowed to continue.

<Item ovf:required="false">
        <rasd:AddressOnParent>0</rasd:AddressOnParent>
        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
        <rasd:ElementName>CD/DVD Drive 1</rasd:ElementName>
        <rasd:InstanceID>7</rasd:InstanceID>
        <rasd:Parent>5</rasd:Parent>
        <rasd:ResourceType>15</rasd:ResourceType>
</Item>

The last item element found in this VirtualHardwareSection is a disk. If we look at the Parent element we can see that we must attach this device to a controller. By looking for InstanceID we can see that this disk's parent is the SCSI controller. Again we specify the AddressOnParent=0 to tell that we want this disk to first on the controller. The HostResource tells what disk defined in the DiskSection we want to use. 'ovf:/disk/vmdisk1' points to an 'ovf:diskId' attribute in the Disk section element. This will tell import process basic information about the disk like capacity and disk format.

<Item>
        <rasd:AddressOnParent>0</rasd:AddressOnParent>
        <rasd:ElementName>Hard Disk 1</rasd:ElementName>
        <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
        <rasd:InstanceID>9</rasd:InstanceID>
        <rasd:Parent>3</rasd:Parent>
        <rasd:ResourceType>17</rasd:ResourceType>
</Item>

We have now defined a small working VM with a SCSI controller, a virtual disk, and a CDROM if supported. Click to download the complete OVF descriptor.

In the next part of this blog post on virtual hardware we will describe more detailed examples of how to create virtual hardware sections.

Delta Disk Support in OVF

OVF packages can become very large when they include several disks.
One way to tackle this issue is by using delta disk compression. It is
a technique that utilizes the fact that many parts of the disks in an
OVF package contain the same data. For example, a collection of VMs in
an OVF package will often run the same kind of operating system.
Generally speaking, delta disk compression arranges a set of disks in a
tree such that components that are equal in child nodes are used in
parent nodes. In this way data is only represented once across multiple
disks.

Here is a conceptual figure of the virtual disks in a delta disk compressed OVF package and how it looks when it is deployed:

DeltaDiskHierachy0
The OVF package contains two VMs: A Web server and a database. They run
the same operating system and delta disk compression has factored out
the common parts in a separate parent disk shared by the two VMs. When
the OVF package is deployed the tree gets flattened and the Web server
and database each get their own copy of the operating system in the
parent disk.

In this blog post you will learn all about how to design your
OVF package to take advantage of delta disks and how to apply this type
of compression to your package using OVF Tool.

  • We start out by looking at a brief example of how a typical OVF
    package with multiple disks could look like and how delta disk
    compression would reduce the size of it.
  • Next we look at what delta disk hierarchies are and how they
    are expressed in the OVF descriptor. This is important to understand
    what they are to make delta disk compression work.
  • Then we give some advice on how you should construct your OVF package to utilize delta disk compression.
    • Here we also give some tips on how to shrink your virtual disks to reduce the space of your OVF package.
  • Finally, we show how OVF Tool can delta disk compress your OVF package.

Example

In the example we look at a multi-tiered LAMP stack.
LAMP is an abbreviation for a software bundle comprising Linux, Apache
HTTP Server, MySQL, and PHP. We split the bundle into two VMs: A Web
server VM running Linux Apache HTTP Server and PHP serving as a
front-end and a database VM running Linux and MySQL as the back-end.
Each VM has a single disk which contains all its data.

Let us assume the two VMs run the same Linux OS (for example Ubuntu
Server 9). Then much of the data on the two disks would be identical
and only the bits concerning the Apache HTTP Server, PHP software, and
MySQL would be different. Here is a rough estimate of how much space
each component will need when stored on a compressed virtual disk:

  • Ubuntu Server 9: 500 MB.
  • Apache HTTP Server and PHP: 50 MB
  • MySQL: 50 MB
Distributing the Web server VM and database VM without delta disk compression would now take up about 1,100 MB:

SizeOf(Web server) + SizeOf(Database) = (500 MB + 50 MB) + (500 MB + 50 MB) = 1,100 MB

In this blog post we will explain how this space can be reduced
using the delta disk feature supported by the OVF specification and OVF
Tool. Using delta disk compression we can extract all the components
that are equal in the two VMs (the Linux OS part), only keeping one
copy of them. This leaves us with an OVF package that only take up
about 600 MB of space.

It is, however, not always as simple as applying delta disk
compression on your OVF package, since it may not yield any reduced
disk space. This is because delta disk compression relies upon how data
is distributed on the disks it works on. In the remainder of the blog
we will explain what delta disks are, how you can optimize your OVF
package to take advantage of delta disk compression, and finally how to
apply delta disk compression to your OVF package with OVF Tool.

Technical Details of Delta Disks

A delta disk hierarchy is a tree of disks like in this figure (white areas denote empty disk space):
DeltaDiskHierachy1

In the figure we see a tree with three nodes: Disk1 (root) with red
data, Disk 2 with blue data and Disk 3 with green data. A disk element
in an OVF descriptor can refer to any of the nodes in the delta disk
hierarchy. For instance, if a disk in the OVF descriptor refers to Disk
3 it will essentially get the flattened Disk 3 shown in the lower half
of the picture when it is deployed. The deployment semantics of a delta
disk node is basically to overlay the nodes in the parent chain
(omitting the white space) from the root all the way down to the chosen
delta disk node. More concretely, in the example to get the flattened
Disk 3, we would first write Disk 1. Then we overwrite this with the
contents of Disk 2 (omitting the empty space) and finally with Disk 3
(omitting the empty space).

In the above paragraph we mention empty space. Empty space is
simply a segment of a disk with containing zeroes, which be a bit
misleading since it may actually used by the VM using the disk.
However, for all intents and purposes it does not matter either way we
look at it.

In the figure parentRefs annotate the arrows that tie
the disks together. This is also what the attribute is called in the
OVF descriptor which link Disk elements together and it is used on Disk
elements in the DiskSection of the OVF descriptor. This is what the
disk section with the three disks could look like:

<DiskSection>
<Info>Meta-information about the virtual disks</Info>
<Disk ovf:capacity="1073741824"
ovf:diskId="disk1"
ovf:fileRef="diskFile1"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
<Disk ovf:capacity="1073741824"
ovf:diskId="disk2"
ovf:fileRef="diskFile2"
ovf:parentRef="disk1"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
<Disk ovf:capacity="1073741824"
ovf:diskId="disk3"
ovf:fileRef="diskFile3"
ovf:parentRef="disk2"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
</DiskSection>

The LAMP example can be described as this delta disk hierarchy:

DeltaDiskHierachy2
For this setup the disk section could look like this:
<DiskSection>
<Info>Meta-information about the virtual disks</Info>
<Disk ovf:capacity="1073741824"
ovf:diskId="parentDisk"
ovf:fileRef="parentDiskFile"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
<Disk ovf:capacity="1073741824"
ovf:diskId="WebServerDisk"
ovf:fileRef="WebServerDiskFile"
ovf:parentRef="parentDisk"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
<Disk ovf:capacity="1073741824"
ovf:diskId="DataBaseDisk"
ovf:fileRef="DatabaseDiskFile"
ovf:parentRef="parentDisk"
ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
</DiskSection>

Preparing an OVF Package for Delta Disk Compression

There are some restrictions of delta disk compression that are
important to understand to get the most out of the feature. Firstly,
the disks in a delta disk hierarchy must have the same capacity, so if
you have two disks in your OVF package with different capacity (for
example, one is 4 GB and the other 8 GB) you will not be able to use
delta disk compression on the two disks. Secondly, delta disk
compression only compares disk content on the same part of the disk at
the disk address level. For example, even though the same file is on
two different disks but not on the same part of the disk it will not be
reduced by delta disk compression. If the file on the first disk is at
address 0×00670000 and on the other disk at address 0x02D10000 it will
not be detected as a shared block and put in a parent disk – only if it
is at the same address (for example, 0×00670000). In other words, for
delta disk compression to work there should be a substantial overlap
between disks at the disk address level.

The second requirement can be difficult to satisfy if you
are not careful in how you construct the OVF package, but there are
ways to do it. To explain how, let us first look at the LAMP stack
example that we looked at in the beginning of the blog post, to see how
we can prepare it for delta disk compression. This LAMP stack had a
Linux VM running Apache HTTP Server and PHP and another Linux VM
running MySQL. Each VM had a single disk.

To achieve optimal disks for delta disk compression we first
create a plain Linux VM with one disk. We clone the VM so we now have
two plain Linux VMs. On one of them we install Apache HTTP Server and
PHP and on the other we install MySQL. By doing this we satisfy the
first criteria that the disks have same capacity (Web server VM’s disk
and database VM’s disk come from the same original plain Linux VM) and
second criteria that a significant part of the disks overlap each
other. The files from the OS part of the plain Linux install are at the
same position on both disks, since they were not changed when we
installed the Apache HTTP Server, PHP, and MySQL (or at least, the
majority of them have not changed).

The above example is rather canonical in how you achieve the
best results from delta disk compression when having multiple VMs using
the same operating system, so to summarize:

  1. Install a plain operating system in a VM;
  2. Clone the plain VM the number of times you need for your solution;
  3. Install the remaining software specific to each VM.
The reason why we first install a plain operating system and then
clone it to the number of VMs we need, rather than installing the same
operating system multiple times on each of the VMs we need, is that we
cannot in general be sure that the files are put at the exact same
location on the disks, even though it is the same operating system.

If cloning is not an option when making the OVF package then
perhaps VMware Studio is. It can create VMs well suited for delta disk
compression, since it builds the VMs operating system and other
software components in a scripted manner that that can be replayed to
produce almost identical VMs.

Shrinking the Disks

When you export your VMs in your OVF package you want to make sure
that all unused space is zeroed out, since this compresses really well
in the VMDK disk format. However, space used by swap disks and deleted
files often take up space on disk, since they are not eagerly zeroed
out by default by most operating systems. This means that even though
your VM says it only uses about 500 MB it may actually take up a lot
more space. Even worse, you may have confidential information on, e.g.,
your swap drive or old deleted files that you do not want to distribute
with the OVF package. There are several ways to solve this problem. On
most Linux distributions it is possible to do the following things to
clean up a disk before you export the VM: 1) Un-mount the swap drive;
2) Write a single file to disk containing only zeroes as large as
possible; 3) Delete the file immediately after you created it. On the
command line you can do these three steps by invoking these commands:

  1. /sbin/swapoff -a (this will un-mount all swap disks)
  2. dd if=/dev/zero of=zeroFile.tmp
  3. rm zeroFile.tmp
On a Windows system it can be done in various ways. We will consider
Windows Server 2008, but it can be applied with modifications on other
types of Windows systems.

We start out by installing VMware tools on the Windows
Server 2008 VM and when it is installed, open VMware tools and choose
“Shrink…”. This will zero out the disk. To zero out the swap disk you
need to set an option under Administrative Tools. Go to Administrative
Tools -> Local Security Policy -> Security Settings ->
Security Options and enable the policy “Shutdown: Clear virtual memory
pagefile”. When you shutdown the VM the swap disk will then be zeroed
out. Please note, however, that enabling this option will increase the
shutdown time significantly for large swap disks. One way of working
around this problem could be to first delete the swap disk, reboot the
VM and disabling the option again (and hopefully no data is written to
the swap disk), and then shutting the VM before putting it in an
OVF package.

Creating an OVF Package with Delta Disk Compression using OVF Tool

Up until now we have not explained how to actually construct an OVF
package with delta disk compression, only what parts go into it. Even
though the ideas behind delta disks can seem a bit complicated it is
quite easy to use delta disk compression in your OVF package by using
OVF Tool. Basically, you use the option –makeDeltaDisks. Source may be
an OVF descriptor, a VMX descriptor, or a VIM source (for example, a VM
or vApp in vSphere). Target must be a directory. For example, we can
use delta disk compression on our LAMP OVF package by invoking this
command:
ovftool --makeDeltaDisks LAMP.ovf output-dir/
This
will create a new OVF package in the output directory with delta disk
compression. That is, both the disk and the OVF descriptor are updated,
which means you can take the OVF package written in the output
directory and deploy it immediately without any manual post processing. There are no restrictions to the type of input you give OVF Tool in
terms of disks. It will try to create delta disk trees of all the disks
in the input OVF package and output the optimal OVF package in terms of
delta disk compression.

The disks that OVF Tool generates are compressed in the
VMDK virtual disk format, but it is possible to apply a second layer of
compression which may yield even smaller disks by using the –compress
option. Use –compress=9 for the best compression. On a package the size
of the LAMP OVF package (about 600 MB) it would yield about 30-40 MB
less disk space (in our experience). Delta disk compressing our LAMP
OVF package with this extra option would then simply be done by
invoking:

ovftool --makeDeltaDisks -compress=9 LAMP.ovf output-dir/
Learn more about what OVF Tool can do by going to the OVF forum at VMware: http://communities.vmware.com/community/developer/forums/ovf. Here you can ask questions about OVF Tool and related products.