VMware

11/06/2009

Virtual Hardware in OVF - Part 1
In the blog post Inside the OVF Package, we talked about the structure of an OVF package, including the different sections (XML elements). One of the more important sections is the one describing the virtual hardware configuration for a VM. The operating system and applications installed in a virtual machine (commonly referred to as just the guest) are highly depended on the virtual hardware that is provided by the deployed virtualization platform. For example, if the hypervisor is exposing the virtual disks through an IDE controller to a guest that is expecting to use a SCSI controller, it is likely that the software will not boot or even worse it might even misbehave. A key objective of the OVF format is to ensure robust and well-behaved deployments. This is achieved by the OVF package author describe the requirements to the virtual hardware in the OVF descriptor, and the OVF deployment can check whether it can fullfill those requirements at deployment time.

In this first blog entry, we will introduce the hardware description model along with a simple examples. This is probably all that you need to know for most daily use. We will follow up with another post with some more examples, and finally with a post with some more advanced examples using deployment-time configurations and multiple hardware profiles.

Ok, lets, get started.

The OVF specification version 1.0 defines a VirtualHardwareSection. This is based on the ResourceAllocationSettingData (RASD) elements from the DMTF CIM standard. For users not familiar with the CIM model this may appear slightly complex at first, but there are many advantages to leveraging an already existing standard, and as we will see once you get used to it, it is pretty straight forward.

The 'System' Element:


We start with the System element found as <System> in the OVF descriptor. The System element is optional but is generally used to specify the “virtual hardware family type”. For VMware products we use the form ‘vmx-X’ where X is VMware's virtual hardware version number. The import process will convert the virtual hardware to use this version number. Several hardware versions may be defined by using space as a delimiter e.g. “vmx-04 vmx-07”. The list of hardware versions are unordered, meaning that the import process is free to pick a hardware version satisfying the remaining of the hardware requirements. VMware products will always pick the recommended hardware version of the host if multiple acceptable versions exists.

The remaining fields are not necessary for import but required by the CIM standard. A system element could look like:

<System>
        <vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
        <vssd:InstanceID>0</vssd:InstanceID>
        <vssd:VirtualSystemIdentifier>MyVm</vssd:VirtualSystemIdentifier>
        <vssd:VirtualSystemType>vmx-07 vmx-4 myHardwareVersion someOtherHardwareVersion</vssd:VirtualSystemType>
</System>

The following table show which hardware versions are supported on various VMware platforms.

Hardware Family Supported by (and later versions of the same products)
vmx-04 WS 5.0, Fusion 1.1, Server 1.0, ESX 3.0
vmx-06 WS 6.0, Fusion 1.1, Server 1.0
vmx-07 WS 6.5, Fusion 2.0, Server 2.0, ESX 4.0

The 'item' Elements

Virtual hardware is modeled as a set of devices, such as ethernet card or disk controllers, memory. Each of those devices are described by an element. The  DMTF RASD specifies a set of fields that can be set. We only a subset of those fields in an OVF descriptor. The fields in a RASD must be ordered alphabetically. The following table lists the RASD fields that are used:


ElementName Required field that contains a display-friendly message about the content of the RASD
Description Human-readable description
InstanceID Required field that contains unique ID within this <VirtualHardwareSection>
ResourceType Required field that indicates the kind of resource
ResourceSubType A vendor-specific identifier for specific devices
VirtualQuantity Specifies the amount (used for memory and CPU)
AllocationUnits Specifies the unit of resource (used for memory and CPU)
Reservation Specifies resource allocation policy (CPU and memory)
Limit Specifies resource allocation policy (CPU and memory)
Weight Specifies resource allocation policy (CPU and memory)
Address Typically used as the unit number of a controller
Parent Instance ID of parent controller (for devices on a controller)
AddressOnParent Used to specify the order for devices on a controller
AutomaticAllocation Used to specify whether a device should be connected on power-on (e.g., for a CDROM)
Connection Reference to a network for an ethernet adaptor
HostResource Reference to a virtual disk for a disk drive

The minimum Item element looks like this:

<Item>
<rasd:ElementName>SomeName</rasd:ElementName>
<rasd:InstanceID>1</rasd:InstanceID>
<rasd:ResourceType>0</rasd:ResourceType>
</Item>

ElementName is a general description of the Item element. InstanceID is a unique identifier for the Item element and is used to refer to other hardware elements. The ResourceType describes the type of hardware. The resource types used in OVF are described in CIM_ResourceAllocationSettingData.mof, which describes the mapping from the ResourceType number to hardware element type. For instance, ResourceType=2 maps to “Processor” (CPU) and ResourceType=10 maps to “Ethernet Adapter”. See the following table for the typical used ResourceTypes: 

Kind ResourceType
Other 0
Processor 3
Memory 4
IDE Controller 5
SCSI Controller 6
Ethernet Adapter 10
Floppy Drive 14
CD/DVD Drive 15/16
Disk Drive 17
USB Controller 23

For some ResourceTypes it is necessary to define a more specific subtype. ResourceSubType is vendor specific, meaning that VMware supports one set of subtypes while other vendors may support other types. ResourceSubType with ResourceType=0 (Other) is also used to define Virtual Hardware that is not described in the CIM schema, for instance a soundcard. The complete list of ResourceTypes with supported ResourceSubTypes for VMware products is shown below:

Kind ResourceType ResourceSubType Other Fields
Other 0 vmware.soundcard.sb16, vmware.soundcard.ensoniq1371, vmware.pcicontroller,vmware.ps2controller, vmware.siocontroller, vmware.keyboard, vmware.pointingdevice Depend on hardware type
Processor 3 AllocationUnit, VirtualQuantity, Reservation, Limit, Weight
Memory 4 AllocationUnit, VirtualQuantity, Reservation, Limit, Weight
IDE Controller 5 Address
SCSI Controller 6 lsilogic, buslogic, lsilogicsas, virtualscsi Address
Ethernet Adapter 10 E1000, PCNet32, VmxNet, VmxNet2, VmxNet3 AddressOnParent, AutomaticAllocation, Connection
Floppy Drive 14 AddressOnParent, AutomaticAllocation
CD/DVD Drive 15 Parent, AddressOnParent, AutomaticAllocation
Disk Drive 17 Parent, AddressOnParent, HostResource
USB Controller 23

Example

We will now look at a sample OVF descriptor that describe a small VM, that are using a single CPU, 512 MB of memory and the use of a disk on a SCSI controller. In this example we will look at the virtual hardware description and will disregard the the other metadata information present in the OVF descriptor. We will look at each of the Item elements in the virtual hardware section and explain what they do and how they reference each other.

The System element specifies that this hardware description is designed for vmx-07 or vmx-04: 

<Info>Virtual hardware requirements</Info>
<System>
        <vssd:ElementName>Virtual Hardware Family</vssd:ElementName>
        <vssd:InstanceID>0</vssd:InstanceID>
        <vssd:VirtualSystemIdentifier>My Small VM</vssd:VirtualSystemIdentifier>
        <vssd:VirtualSystemType>vmx-07 vmx-04</vssd:VirtualSystemType>
</System>

If we look at the two first Item elements we first see a definition of how many CPUs we want the VM to have. The amount is in the VirtualQuantity element and for this VM we want 1 CPU. In AllocationUnits we are using the Programmatic Units as defined in http://www.dmtf.org/standards/published_documents/DSP0004_2.5.0.pdf. The next Item element we are defining the amount of memory.

<Item>
        <rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits>
        <rasd:Description>Number of Virtual CPUs</rasd:Description>
        <rasd:ElementName>1 virtual CPU(s)</rasd:ElementName>
        <rasd:InstanceID>1</rasd:InstanceID>
        <rasd:ResourceType>3</rasd:ResourceType>
        <rasd:VirtualQuantity>1</rasd:VirtualQuantity>
</Item>
<Item>
        <rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits>
        <rasd:Description>Memory Size</rasd:Description>
        <rasd:ElementName>512MB of memory</rasd:ElementName>
        <rasd:InstanceID>2</rasd:InstanceID>
        <rasd:ResourceType>4</rasd:ResourceType>
        <rasd:VirtualQuantity>512</rasd:VirtualQuantity>
</Item>

We now define a SCSI controller of subtype ‘lsilogic’ and with an InstanceID=3. We will later use the InstanceID to bind a disk to this controller. As can be seen we have not defined a PCI controller to bind the SCSI controller to. For VMware products PCI controller is a default device so there it is optional to add it as an Item element. The next two Item elements are defining IDE controllers.

<Item>
        <rasd:Address>0</rasd:Address>
        <rasd:Description>SCSI Controller</rasd:Description>
        <rasd:ElementName>SCSI Controller 0</rasd:ElementName>
        <rasd:InstanceID>3</rasd:InstanceID>
        <rasd:ResourceSubType>lsilogic</rasd:ResourceSubType>
        <rasd:ResourceType>6</rasd:ResourceType>
</Item>
<Item>
        <rasd:Address>1</rasd:Address>
        <rasd:Description>IDE Controller</rasd:Description>
        <rasd:ElementName>IDE 1</rasd:ElementName>
        <rasd:InstanceID>4</rasd:InstanceID>
        <rasd:ResourceType>5</rasd:ResourceType>
</Item>
<Item>
        <rasd:Address>0</rasd:Address>
        <rasd:Description>IDE Controller</rasd:Description>
        <rasd:ElementName>IDE 0</rasd:ElementName>
        <rasd:InstanceID>5</rasd:InstanceID>
        <rasd:ResourceType>5</rasd:ResourceType>
</Item>

We now want to add a CDROM drive to one of the IDE controllers. If we look at the Parent element we can see we want to bind it to a device with InstanceID=5. If we look for an Item element with InstanceID=5 we can see that this is IDE 0. We use AddressOnParent to tell where on the IDE controller we want to place the CDROM. If no AddressOnParent element is specified devices are attached to their “parent” in the order they are found in the OVF. Here we see that the AddressOnParent=0, so we set it to device 0 on the IDE controller. On the Item element we are defining a ovf:required=”false” this tells the import process that if we are unable to add this hardware device the import process is allowed to continue.

<Item ovf:required="false">
        <rasd:AddressOnParent>0</rasd:AddressOnParent>
        <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation>
        <rasd:ElementName>CD/DVD Drive 1</rasd:ElementName>
        <rasd:InstanceID>7</rasd:InstanceID>
        <rasd:Parent>5</rasd:Parent>
        <rasd:ResourceType>15</rasd:ResourceType>
</Item>

The last item element found in this VirtualHardwareSection is a disk. If we look at the Parent element we can see that we must attach this device to a controller. By looking for InstanceID we can see that this disk's parent is the SCSI controller. Again we specify the AddressOnParent=0 to tell that we want this disk to first on the controller. The HostResource tells what disk defined in the DiskSection we want to use. 'ovf:/disk/vmdisk1' points to an 'ovf:diskId' attribute in the Disk section element. This will tell import process basic information about the disk like capacity and disk format.

<Item>
        <rasd:AddressOnParent>0</rasd:AddressOnParent>
        <rasd:ElementName>Hard Disk 1</rasd:ElementName>
        <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
        <rasd:InstanceID>9</rasd:InstanceID>
        <rasd:Parent>3</rasd:Parent>
        <rasd:ResourceType>17</rasd:ResourceType>
</Item>

We have now defined a small working VM with a SCSI controller, a virtual disk, and a CDROM if supported. Click to download the complete OVF descriptor.

In the next part of this blog post on virtual hardware we will describe more detailed examples of how to create virtual hardware sections.

Posted by Eske Christiansen on November 6, 2009 | Permalink | Comments (0) | TrackBack (0) |

10/16/2009

Delta Disk Support in OVF
OVF packages can become very large when they include several disks. One way to tackle this issue is by using delta disk compression. It is a technique that utilizes the fact that many parts of the disks in an OVF package contain the same data. For example, a collection of VMs in an OVF package will often run the same kind of operating system. Generally speaking, delta disk compression arranges a set of disks in a tree such that components that are equal in child nodes are used in parent nodes. In this way data is only represented once across multiple disks.

Here is a conceptual figure of the virtual disks in a delta disk compressed OVF package and how it looks when it is deployed:

DeltaDiskHierachy0
The OVF package contains two VMs: A Web server and a database. They run the same operating system and delta disk compression has factored out the common parts in a separate parent disk shared by the two VMs. When the OVF package is deployed the tree gets flattened and the Web server and database each get their own copy of the operating system in the parent disk.

In this blog post you will learn all about how to design your OVF package to take advantage of delta disks and how to apply this type of compression to your package using OVF Tool.

  • We start out by looking at a brief example of how a typical OVF package with multiple disks could look like and how delta disk compression would reduce the size of it.
  • Next we look at what delta disk hierarchies are and how they are expressed in the OVF descriptor. This is important to understand what they are to make delta disk compression work.
  • Then we give some advice on how you should construct your OVF package to utilize delta disk compression.
    • Here we also give some tips on how to shrink your virtual disks to reduce the space of your OVF package.
  • Finally, we show how OVF Tool can delta disk compress your OVF package.

Example

In the example we look at a multi-tiered LAMP stack. LAMP is an abbreviation for a software bundle comprising Linux, Apache HTTP Server, MySQL, and PHP. We split the bundle into two VMs: A Web server VM running Linux Apache HTTP Server and PHP serving as a front-end and a database VM running Linux and MySQL as the back-end. Each VM has a single disk which contains all its data.

Let us assume the two VMs run the same Linux OS (for example Ubuntu Server 9). Then much of the data on the two disks would be identical and only the bits concerning the Apache HTTP Server, PHP software, and MySQL would be different. Here is a rough estimate of how much space each component will need when stored on a compressed virtual disk:

  • Ubuntu Server 9: 500 MB.
  • Apache HTTP Server and PHP: 50 MB
  • MySQL: 50 MB
Distributing the Web server VM and database VM without delta disk compression would now take up about 1,100 MB:

SizeOf(Web server) + SizeOf(Database) = (500 MB + 50 MB) + (500 MB + 50 MB) = 1,100 MB

In this blog post we will explain how this space can be reduced using the delta disk feature supported by the OVF specification and OVF Tool. Using delta disk compression we can extract all the components that are equal in the two VMs (the Linux OS part), only keeping one copy of them. This leaves us with an OVF package that only take up about 600 MB of space.

It is, however, not always as simple as applying delta disk compression on your OVF package, since it may not yield any reduced disk space. This is because delta disk compression relies upon how data is distributed on the disks it works on. In the remainder of the blog we will explain what delta disks are, how you can optimize your OVF package to take advantage of delta disk compression, and finally how to apply delta disk compression to your OVF package with OVF Tool.

Technical Details of Delta Disks

A delta disk hierarchy is a tree of disks like in this figure (white areas denote empty disk space):
DeltaDiskHierachy1

In the figure we see a tree with three nodes: Disk1 (root) with red data, Disk 2 with blue data and Disk 3 with green data. A disk element in an OVF descriptor can refer to any of the nodes in the delta disk hierarchy. For instance, if a disk in the OVF descriptor refers to Disk 3 it will essentially get the flattened Disk 3 shown in the lower half of the picture when it is deployed. The deployment semantics of a delta disk node is basically to overlay the nodes in the parent chain (omitting the white space) from the root all the way down to the chosen delta disk node. More concretely, in the example to get the flattened Disk 3, we would first write Disk 1. Then we overwrite this with the contents of Disk 2 (omitting the empty space) and finally with Disk 3 (omitting the empty space).

In the above paragraph we mention empty space. Empty space is simply a segment of a disk with containing zeroes, which be a bit misleading since it may actually used by the VM using the disk. However, for all intents and purposes it does not matter either way we look at it.

In the figure parentRefs annotate the arrows that tie the disks together. This is also what the attribute is called in the OVF descriptor which link Disk elements together and it is used on Disk elements in the DiskSection of the OVF descriptor. This is what the disk section with the three disks could look like:

<DiskSection>
 <Info>Meta-information about the virtual disks</Info>
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="disk1" 
       ovf:fileRef="diskFile1" 
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="disk2" 
       ovf:fileRef="diskFile2" 
      ovf:parentRef="disk1"
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="disk3" 
       ovf:fileRef="diskFile3" 
       ovf:parentRef="disk2"
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
</DiskSection>

The LAMP example can be described as this delta disk hierarchy:

DeltaDiskHierachy2
For this setup the disk section could look like this:
<DiskSection>
 <Info>Meta-information about the virtual disks</Info>
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="parentDisk" 
       ovf:fileRef="parentDiskFile" 
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="WebServerDisk" 
       ovf:fileRef="WebServerDiskFile" 
      ovf:parentRef="parentDisk"
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized"/>
 <Disk ovf:capacity="1073741824" 
       ovf:diskId="DataBaseDisk" 
       ovf:fileRef="DatabaseDiskFile" 
       ovf:parentRef="parentDisk"
       ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" />
</DiskSection>

Preparing an OVF Package for Delta Disk Compression

There are some restrictions of delta disk compression that are important to understand to get the most out of the feature. Firstly, the disks in a delta disk hierarchy must have the same capacity, so if you have two disks in your OVF package with different capacity (for example, one is 4 GB and the other 8 GB) you will not be able to use delta disk compression on the two disks. Secondly, delta disk compression only compares disk content on the same part of the disk at the disk address level. For example, even though the same file is on two different disks but not on the same part of the disk it will not be reduced by delta disk compression. If the file on the first disk is at address 0x00670000 and on the other disk at address 0x02D10000 it will not be detected as a shared block and put in a parent disk – only if it is at the same address (for example, 0x00670000). In other words, for delta disk compression to work there should be a substantial overlap between disks at the disk address level.

The second requirement can be difficult to satisfy if you are not careful in how you construct the OVF package, but there are ways to do it. To explain how, let us first look at the LAMP stack example that we looked at in the beginning of the blog post, to see how we can prepare it for delta disk compression. This LAMP stack had a Linux VM running Apache HTTP Server and PHP and another Linux VM running MySQL. Each VM had a single disk.

To achieve optimal disks for delta disk compression we first create a plain Linux VM with one disk. We clone the VM so we now have two plain Linux VMs. On one of them we install Apache HTTP Server and PHP and on the other we install MySQL. By doing this we satisfy the first criteria that the disks have same capacity (Web server VM’s disk and database VM’s disk come from the same original plain Linux VM) and second criteria that a significant part of the disks overlap each other. The files from the OS part of the plain Linux install are at the same position on both disks, since they were not changed when we installed the Apache HTTP Server, PHP, and MySQL (or at least, the majority of them have not changed).

The above example is rather canonical in how you achieve the best results from delta disk compression when having multiple VMs using the same operating system, so to summarize:

  1. Install a plain operating system in a VM;
  2. Clone the plain VM the number of times you need for your solution;
  3. Install the remaining software specific to each VM.
The reason why we first install a plain operating system and then clone it to the number of VMs we need, rather than installing the same operating system multiple times on each of the VMs we need, is that we cannot in general be sure that the files are put at the exact same location on the disks, even though it is the same operating system.

If cloning is not an option when making the OVF package then perhaps VMware Studio is. It can create VMs well suited for delta disk compression, since it builds the VMs operating system and other software components in a scripted manner that that can be replayed to produce almost identical VMs.

Shrinking the Disks

When you export your VMs in your OVF package you want to make sure that all unused space is zeroed out, since this compresses really well in the VMDK disk format. However, space used by swap disks and deleted files often take up space on disk, since they are not eagerly zeroed out by default by most operating systems. This means that even though your VM says it only uses about 500 MB it may actually take up a lot more space. Even worse, you may have confidential information on, e.g., your swap drive or old deleted files that you do not want to distribute with the OVF package. There are several ways to solve this problem. On most Linux distributions it is possible to do the following things to clean up a disk before you export the VM: 1) Un-mount the swap drive; 2) Write a single file to disk containing only zeroes as large as possible; 3) Delete the file immediately after you created it. On the command line you can do these three steps by invoking these commands:

  1. /sbin/swapoff -a (this will un-mount all swap disks)
  2. dd if=/dev/zero of=zeroFile.tmp
  3. rm zeroFile.tmp
On a Windows system it can be done in various ways. We will consider Windows Server 2008, but it can be applied with modifications on other types of Windows systems.

We start out by installing VMware tools on the Windows Server 2008 VM and when it is installed, open VMware tools and choose “Shrink…”. This will zero out the disk. To zero out the swap disk you need to set an option under Administrative Tools. Go to Administrative Tools -> Local Security Policy -> Security Settings -> Security Options and enable the policy “Shutdown: Clear virtual memory pagefile”. When you shutdown the VM the swap disk will then be zeroed out. Please note, however, that enabling this option will increase the shutdown time significantly for large swap disks. One way of working around this problem could be to first delete the swap disk, reboot the VM and disabling the option again (and hopefully no data is written to the swap disk), and then shutting the VM before putting it in an OVF package.

Creating an OVF Package with Delta Disk Compression using OVF Tool

Up until now we have not explained how to actually construct an OVF package with delta disk compression, only what parts go into it. Even though the ideas behind delta disks can seem a bit complicated it is quite easy to use delta disk compression in your OVF package by using OVF Tool. Basically, you use the option –makeDeltaDisks. Source may be an OVF descriptor, a VMX descriptor, or a VIM source (for example, a VM or vApp in vSphere). Target must be a directory. For example, we can use delta disk compression on our LAMP OVF package by invoking this command:
ovftool --makeDeltaDisks LAMP.ovf output-dir/
This will create a new OVF package in the output directory with delta disk compression. That is, both the disk and the OVF descriptor are updated, which means you can take the OVF package written in the output directory and deploy it immediately without any manual post processing. There are no restrictions to the type of input you give OVF Tool in terms of disks. It will try to create delta disk trees of all the disks in the input OVF package and output the optimal OVF package in terms of delta disk compression.

The disks that OVF Tool generates are compressed in the VMDK virtual disk format, but it is possible to apply a second layer of compression which may yield even smaller disks by using the –compress option. Use –compress=9 for the best compression. On a package the size of the LAMP OVF package (about 600 MB) it would yield about 30-40 MB less disk space (in our experience). Delta disk compressing our LAMP OVF package with this extra option would then simply be done by invoking:

ovftool --makeDeltaDisks -compress=9 LAMP.ovf output-dir/
Learn more about what OVF Tool can do by going to the OVF forum at VMware: http://communities.vmware.com/community/developer/forums/ovf. Here you can ask questions about OVF Tool and related products.

Posted by Kristian Bisgaard Lassen on October 16, 2009 in Web/Tech | Permalink | Comments (0) | TrackBack (0) |

09/30/2009

VMware Studio - Adding RHEL 5.4 support in minutes.

Red Hat has recently released Red Hat Enterprise Linux (RHEL) 5.4. Here are the easy steps for enabling VMware Studio 2.0 to create VM or vApp for RHEL 5.4 Server i386 and x86_64.

Adding RHEL 5.4 i386 support structure based on the existing RHEL 5.3 i386 components:

  1. Log in to the Studio VM as root.
  2. Create the template profile, Studio runtime packages, and VMware Tools for the new OS:
    • cp -r /opt/vmware/etc/build/templates/redhat/5/3 /opt/vmware/etc/build/templates/redhat/5/4
    • cp -r /opt/vmware/lib/build/include/rhel/5/3 /opt/vmware/lib/build/include/rhel/5/4
    • cp -r /opt/vmware/www/vmware-open-vm-tools/redhat/5/3 /opt/vmware/www/vmware-open-vm-tools/redhat/5/4
  3. Edit and update /opt/vmware/etc/build/templates/redhat/5/4/build_profile.xml with the following content:
    • [line 25] <Product>Red Hat Enterprise Linux 5.4</Product>
    • [line 92] <vadk:url>http://[VADK.localIP]/vmware-open-vm-tools/redhat/5/4</vadk:url>
    • [line 208] <Description>Red Hat Enterprise Linux 5.4</Description>
    • [line 303] vadk:sourceDir="[VADK.vadkRoot]/lib/build/include/rhel/5/4/"
    • [line 306] <vadk:ISO vadk:path="file:///opt/vmware/www/ISV/ISO/rhel-server-5.4-i386-dvd.iso"
          vadk:md5sum="7a12ec6599527e4f3d1790b51eadbfed" vadk:containFiles=""/>
    • [line 309] <vadk:Distribution vadk:vendor="Red Hat" vadk:OSverMajor="5" vadk:OSverMinor="4"

Adding RHEL 5.4 x86_64 support structure based on the existing RHEL 5.3 x86_64 components:

  1. Log in to the Studio VM as root.
  2. Create the template profile, Studio runtime packages, and VMware Tools for the new OS:
    • cp -r /opt/vmware/etc/build/templates/redhat/5/3_x86_64 /opt/vmware/etc/build/templates/redhat/5/4_x86_64
    • cp -r /opt/vmware/lib/build/include/rhel/5/3_x86_64 /opt/vmware/lib/build/include/rhel/5/4_x86_64
    • cp -r /opt/vmware/www/vmware-open-vm-tools/redhat/5/3_x86_64 /opt/vmware/www/vmware-open-vm-tools/redhat/5/4_x86_64
  3. Edit and update /opt/vmware/etc/build/templates/redhat/5/4_x86_64/build_profile.xml with the following content:
    • [line 25] <Product>Red Hat Enterprise Linux 5.4 64bit</Product>
    • [line 91] <vadk:url>http://[VADK.localIP]/vmware-open-vm-tools/redhat/5/4_x86_64</vadk:url>
    • [line 208] <Description>Red Hat Enterprise Linux 5.4 64bit</Description>
    • [line 303] vadk:sourceDir="[VADK.vadkRoot]/lib/build/include/rhel/5/4_x86_64/"
    • [line 305] <vadk:ISO vadk:path="file:///opt/vmware/www/ISV/ISO/rhel-server-5.4-x86_64-dvd.iso"
          vadk:md5sum="04fe3c10202402d7b389528d2bad0210" vadk:containFiles=""/>
    • [line 308] <vadk:Distribution vadk:vendor="Red Hat" vadk:OSverMajor="5" vadk:OSverMinor="4"

And that's it. You will now see the RHEL 5.4 template profiles through the VMware Studio web interface. With the new template profiles, you may then create your customized build profile for building the desired RHEL 5.4 Server VM or vApp.

Technorati Tags: , , , , , , , ,


Posted by Leong on September 30, 2009 in Web/Tech | Permalink | Comments (2) | TrackBack (0) |

09/22/2009

VMware Studio 2.0 GA - What's new?
VMware Studio 2.0 GA was announced at VMworld Developer Day. The GA release comes two months after Beta. It mainly focuses on product quality and usability improvements with few enhancements from the Beta feedback. One of the key features added was support for CD based updates for virtual appliances. This feature is critical for production appliances that are deployed behind firewall. Since these appliances cannot connect to ISV's web based update repository, GA version of Studio allows appliance authors to create CD based update repository alongwith web based version it supported since 1.0. The end-user web console allows users to configure updates to be received from CD or through the phone-home URL. The other key area where we spent a good time during GA release was ensuring support for various vCenter environments for building vApps and virtual appliances. You can have VC environments with distributed vSwitches, VC deployments integrated with Active Directory and specify them as provisioning target in Studio. GA version also allows the author to specify clusters and sub resource pools for provisioning hosts. It detects hosts that are in maintenance mode and prevents their use during provisioning. 

Minor features include support for web console footer customization, setting hostname explicitly in built vms and virtual appliances and control EULA acceptance on first boot amongst others. Disabling EULA acceptance is needed for build and test environments to enable running regression test suite on nightly appliance builds in automated fashion. With 2.0 GA release, VMware tools are optional and explicitly specified in the build profile similar to appliance packages. This makes it easy to update or replace the tools with the version preferred for your deployment environment.

Also check out the GA version of Eclipse Plugin. We have done few usability improvements here, ability to add individual packages instead of entire application package repositories, ability to automatically create application package repository when building an application package. Studio menu items in Eclipse are now context sensitive. VMware Studio 2.0 is a view in Eclipse. Try it out with your Java, J2EE, C++ perspectives and it should integrate seamlessly. Eclipse 3.5 was released just after Studio Beta and we have ensured Studio GA Plugin continues to work with 3.5 alongwith 3.4 version of Eclipse.

A common question seen in community forums is support for various guest Operating Systems. We plan to address it in a different form by making Studio build process agnostic of various Linux distros and enabling users to create their own templates. Expect this feature in near future. In the meanwhile we have various blog posts coming to describe how to create VMs with various guest operating systems that are not supported out of the box in VMware Studio. 

The GA version includes fixes that we encountered while creating large vApps. One of the exercises we conducted internally was creation of Sharepoint vApp using Studio. We successfully created the 3 tier vApp that has MSSQL database, Index server and Frontend. The Sharepoint vApp uses OVF properties to extract environment specific variables : computer names of each vm and domain to integrate into. On first boot it runs sysprep to complete the deployment process. It also asks for existing AD instance and integrates the vApp into the existing domain. The end result is a running sharepoint vApp integrated with AD. We plan to post the details around the same including first boot scripts, sysprep commands, etc in coming weeks. 

Below are part 1 and 2 of videos that show how to build vApps using VMware Studio. Download VMware Studio 2.0 GA release from here and give it a spin. We look forward for your feedback.

--From Studio Team

   
 

Technorati Tags: , , , ,


Posted by Komal Mangtani on September 22, 2009 in Web/Tech | Permalink | Comments (0) | TrackBack (0) |

09/11/2009

OVF Localization
OVF packages are ideal for distributing complex software packages. As we saw in the post on "Inside the OVF Package", it is possible to tailor the same OVF package for different hypervisors using deployment options. Moreover, it is possible to add meaningful human readable descriptions to the different parts of the OVF descriptor, in places like product information, EULA sections and deployment options. This enables deployment wizards (like the vSphere client) to give the user a great experience when deploying an OVF package, since it can use specific messages relevant to a particular OVF descriptor. For example, the user can see messages about the intention of deployment options and properties, and how to use them. This blog post is about what to do about all that human readable metadata, when you want to distribute your OVF package in different countries and for different languages.

The OVF specification includes an internationalization section that describes how to localize an OVF descriptor. This lets you address the issue of translating all the human readable metadata in the OVF descriptor without having to keep multiple copies of the descriptor, each localized to a specific language. Obviously it does not make sense to localize everything in an OVF descriptor, since some information is only intended to be read by a machine. A rule of thumb is that you can localize all the elements that carry some kind of metadata which are useful to the deployer of the OVF package but not needed by the deployment platform. For a full list of elements that can be localized please see the list at the end of this blog post.

Example

Let us take a look at how the user experiences an OVF package that has been localized:

OVF deployment using default language.

Ovfdeploy_english 

OVF deployment using German localization.

Ovfdeploy_german

The vSphere Client attempts to use the localized messages from the OVF package which matches the locale of the users Windows installation. If a matching localization is not found, the default language of the OVF descriptor (English in the example) is used.

Localizing the OVF Descriptor

Let’s look at how to write an OVF descriptor that is localized to multiple language including both English and German as in the above example, but also Danish and Swahili:

<Envelope xml:lang="en-US">
  ...
  <VirtualSystem ovf:id="MyVM">
    ...
    <ProductSection>
      <Info>Information about the installed software</Info>
      <Property ovf:key="num_connections" ovf:type="string"
        ovf:userConfigurable="true">
        <Label ovf:msgid="num_connections.label">
          Number of connections
        </Label>
      </Property>
      <Property ovf:key="admin_address" ovf:type="string"
        ovf:userConfigurable="true">
        <Label ovf:msgid="admin_address.label">Administrator address</Label>
        <Description ovf:msgid="admin_address.description">
      Email address of the systems administrator
    </Description>
      </Property>
    </ProductSection>
    ...
  </VirtualSystem>
 
  <!-- German localized messages -->
  <Strings xml:lang="de-DE">
    <Msg ovf:msgid="num_connections.label">Zahl der Anschlüsse</Msg>
    <Msg ovf:msgid="admin_address.label">Verwalteradresse</Msg>
    <Msg ovf:msgid="admin_address.description">
      Email address des Systemverwalters
    </Msg>
  </Strings>
 
   <!-- Danish localized messages -->
    <Strings xml:lang="da-DK">
    <Msg ovf:msgid="num_connections.label">Antal forbindelser</Msg>
    <Msg ovf:msgid="admin_address.label">Administrator adresse</Msg>
    <Msg ovf:msgid="admin_address.description">
      System administratorens email-adresse
    </Msg>
  </Strings>
 
   <!-- Swahili localized messages -->
    <Strings xml:lang="sw">
    <Msg ovf:msgid="num_connections.label">Idadi ya connections</Msg>
    <Msg ovf:msgid="admin_address.label">Administrator anwani</Msg>
    <Msg ovf:msgid="admin_address.description">
      Barua pepe ya system administrator
    </Msg>
  </Strings>
</Envelope>

In this example, we are localizing the label of the "num_connections" property and the label and description of the "admin_address" property.

As you can see, it is pretty straight forward to localize an OVF descriptor to support multiple languages:

  1. First prepare the OVF descriptor for localization by adding an ovf:msgid attribute to each of the elements you want to be localized (see the list of possible elements in the last section of this blog entry) and give the ovf:msgid a unique value;
  2. Next you add a Strings section for each of the locales you want to support and specify the locale by using the xml:lang attribute;
  3. For each ovf:msgid attribute in the OVF descriptor create a Msg element in the Strings section with the same ovf:msgid value.

The human readable messages in the example ProductSection element are used as the default language and will be used if no appropriate locale is available in the descriptor. To specify the default language of the OVF descriptor (the language used in the default messages), set the xml:lang attribute on the top Envelope element level. In our example we have set it to US English (en-US).

The format of the locale is standard and is written as [language]-[country]-[variant]. It is not necessary to specify the full locale for a string bundle. For example, if we simply specify the German locale in the example as de it can then be used for several German speaking countries such as Austria (de-AT), Germany (de-DE), Luxembourg (de-LU) and parts of Switzerland (de-CH). If a user uses locale X on his computer, we select the locale in the OVF descriptor which matches the beginning or all of X. If two locales match X we chose the one that gives the longest match. This is why changing the locale from de-DE to de means that the localization can be used on the locales de-AT, de-DE, de-LU and de-CH, since de is a prefix of those.

External String Bundles

Often when managing multiple locales it is impractical to keep them in the same file. For this purpose the OVF specification allows you to extract all the Strings elements and put them in separate files. From the example OVF descriptor we have used, we can take all the three different localizations and put them in separate files (we will call these files string bundles):

<Envelope>
  <References>
    <File ovf:href="MyVM-de-DE.msg" ovf:id="msgs1"/>
    <File ovf:href="MyVM-da-dk.msg" ovf:id="msgs2"/>
    <File ovf:href="MyVM-sw.msg" ovf:id="msgs3"/>
    <File ovf:href="MyVM-disk1.vmdk" ovf:id="file2" ovf:size="9580544"/>
    ...
  </References>
  ...
</Envelope>

The files are then referenced in the beginning of the Reference section in the OVF descriptor (important: String bundles must be listed first in the reference section):

File: MyVM-de-DE.msg

<!-- German localized messages -->
<Strings xml:lang="de-DE">
  <Msg ovf:msgid="num_connections.label">Zahl der Anschlüsse</Msg>
  <Msg ovf:msgid="admin_address.label">Verwalteradresse</Msg>
  <Msg ovf:msgid="admin_address.description">
    Email address des Systemverwalters
  </Msg>
</Strings>
 

File: MyVM-da-DK.msg

<!-- Danish localized messages -->
<Strings xml:lang="da-DK">
  <Msg ovf:msgid="num_connections.label">Antal forbindelser</Msg>
  <Msg ovf:msgid="admin_address.label">Administrator adresse</Msg>
  <Msg ovf:msgid="admin_address.description">
      System administratorens email-adresse
  </Msg>
</Strings>
 

File: MyVM-sw.msg


<!-- Swahili localized messages -->
<Strings xml:lang="sw">
  <Msg ovf:msgid="num_connections.label">Idadi ya connections</Msg>
  <Msg ovf:msgid="admin_address.label">Administrator anwani</Msg>
  <Msg ovf:msgid="admin_address.description">
    Barua pepe ya system administrator
  </Msg>
</Strings>

In the above example we created three files, one for each locale. The OVF specification allows multiple Strings elements in the same file next to the OVF descriptor, so it is not necessary to create a file per locale as we did. However, by keeping the locales in separate string bundles it become easy to extend the supported locales simply by adding more string bundle files.

Further Reading

To learn more about localization, check out section in the OVF 1.0.0 specification: http://www.dmtf.org/standards/published_documents/DSP0243_1.0.0.pdf. 

List of Localizable Elements

The text in the following elements can be localized:
  • Info element on VirtualSystem and VirtualSystemCollection
  • Name element on VirtualSystem and VirtualSystemCollection
  • Info element on AnnotationSection, DeploymentOptionSection, DiskSection, EulaSection, InstallSection, NetworkSection, OperatingSystemSection, ProductSection, ResourceAllocationSection, StartupSection and VirtualHardwareSection.
  • Annotation element on AnnotationSection
  • License element on EulaSection
  • Description element on NetworkSection
  • Description element on OperatingSystemSection
  • Description, Product, Vendor, Label, and Category elements on ProductSection
  • Description and Label elements on DeploymentOptionSection
  • ElementName, Caption and Description sub-elements on the System element in VirtualHardwareSection
  • ElementName, Caption and Description sub-elements on Item elements in VirtualHardwareSection
  • ElementName, Caption and Description sub-elements on Item elements in ResourceAllocationSection



Posted by Kristian Bisgaard Lassen on September 11, 2009 in Web/Tech | Permalink | Comments (0) | TrackBack (0) |

08/19/2009

Configuring OVF Properties through VMware Studio 2.0

Previous blog post "Self-Configuration and the OVF Environment" introduced the concept of OVF Environment and how an author of a VM or a vApp can provide deployment options to configure a VM or a vApp using OVF properties. These OVF properties can be common to all VMs, for instance IP configuration, and others which are application specific. For example: An application which has a notification mechanism may require an email address, OVF properties can be used to ask the deployer of the VM to enter the administrator’s email address.

Studio supports creation of user defined application specific OVF properties, creates network related properties for VM by default, and provides an in-guest agent (glue code) for Linux VMs to configure the network information for the VM.

Application specific OVF properties specified for a VM/vApp in Studio requires an in-guest script which can consume these properties. This script would parse the key/value pairs of the OVF environment and configure the application accordingly. This script can be installed as a package (rpm/deb/vsp) in the VM and invoked as a part of the boot customization (firstboot script) provided in Studio.

 In order to see Studio’s support for OVF properties in action, OVF 1.0 VM/vApps (or OVA) generated by Studio 2.0 and vCenter 4.0 is required. OVF environment is not present in the guest OS while deploying on ESX 4.0 directly or if OVF 0.9 is being used.

IP related OVF properties

The previous blog entry gave a quick overview of defining static IP properties and using these IP related properties to configure your VM. Studio provides support for network properties out of the box.

With Studio you can:

  • Allow users to choose either DHCP or Static networking for VM/vApps during deployment.
  • Allow users to select Transient IPs (or a pool of IPs which can be configured in vSphere and used as an alternative to Static IPs)
  • Allows authors to restrict the IP Allocation policy to either DHCP or Static/Transient, or all three.
  • Configure networking for all Linux VMs, even those which are part of a vApp.
  • Reduce the amount of information that needs to be asked. Studio uses dynamic properties, so that entering common network information like subnet information; gateway etc can be avoided for each VM.

In the Output tab of the Studio build wizard, the OVF IP Allocation/Assignment Scheme determines if the user will be allowed to use Static or DHCP networking for the application. If the OVF environment option is checked then the user is presented with a choice of using Static or Transient IPs during deployment. If DHCP is the only IP Allocation Scheme selected then the user is not prompted to specify information for any networking related OVF property generated by Studio 2.0.

When the user selects Static IP during deployment, the user is prompted to enter only the IP address for that VM. The rest of the information to configure the networking, the subnet mask, gateway, dns servers are retrieved from a construct in vCenter 4.0 called IP Pools. To create an IP Pool click on the datacenter in vSphere Client where the VM is being deployed and select the IP Pool tab, now click on Add link in the tab. Here specify the IPv4 subnet, gateway, and optionally DNS servers. An IP pool is associated with networks, during deployment of a VM a network should be selected, ensure that this IP pool is associated with that network.

Transient IPs are similar to Static IPs but the user doesn’t have to enter an IP address. The address is retrieved from a range of IP addresses from IP pools in vCenter. In the IP pool configuration click on Enable IP pool, and enter ranges of IPs that are reserved for VMs. vCenter manages the IPs by allocating them on demand.

If there is a DHCP server on the network then specifying the one exists on the network, in the IP pool configuration is recommended.

When you build a vApp from VMs which were created using Studio 2.0, the networking properties are re-configured, so that all the VMs in the vApp have valid networking properties.

Other OVF properties

Additional OVF properties automatically supported by VMware Studio that you may find useful are:

  • vm.name property which contains the name of the VM, this is a VM specific property. It is useful in vApps, each VM can identify itself based on this property. This property is present in all OVFs generated by Studio.
  • vami.timezone: Studio provided in-guest agent configures timezone for a Linux VM if it sees this property have a valid timezone string. This property is available in the default template for vApps. For VMs, it needs to be explicitly added to the OVF Properties section in Output tab.
  • vami.hostname:  This property is used to configure the hostname for a Linux VM. For those VMs which are going to be configured with Static IP and require a valid hostname to be specified, it is recommended that this property be used to specify a hostname. This property needs to be explicitly added to the OVF Properties section in Output tab. (not supported in Studio 2.0 beta).

Accessing OVF Enviroment

Studio packages VMware Tools for all VMs that are built, as a result the transport of OVF environment to the guest uses VMware Tools. The OVF environment can be accessed within Linux VMs using either the command

/opt/vmware/share/vami/vami_ovf_process --printovfenv

Or by simply looking up file:

/opt/vmware/etc/vami/ovfEnv.xml (not present in Studio 2.0 beta)

Other possible uses

OVF properties can be broadly classified into two types from a deployer’s perspective:

User Configurable

This is best used to prompt user for inputs and configure the application. To make a property user configurable make sure that the user configurable option is set to true when defining a new property.

Non-User Configurable

Some OVF properties are dynamic in nature, i.e. the values for these properties are provided by vCenter dynamically. In order to specify such properties a valid vmw: tag must be specified, this can be achieved by defining a property as custom datatype. Example: To access vCenter IP Address you can use the vmw qualifier VimIp("").

Other class of non-user configurable OVF properties is static properties. They are similar to user configurable properties in all aspects except the user will not be prompted to enter values during deployment. They can be used to control application behavior. For instance, if you want to control the verbosity of the application log or set values for application timers to control application behavior on various deployment scenarios.

More links:
VMware Studio 2.0 Overview
VMware Studio 2.0 Beta webinar recording
VMware Studio 2.0 Beta podcast

Posted by Vishwas Nagaraja on August 19, 2009 | Permalink | Comments (0) | TrackBack (0) |

08/10/2009

Inside the OVF Package

In the previous blog posts we have talked how OVF packages can be deployed using the vSphere client and OVF Tool, and how to use OVF to create self-configurable virtual machine templates. In this blog post we will look inside an OVF package to see how it is structured and organized.

OVF is an open standard developed by the Distributed Management Task Force (DMTF) with cooperation from VMware, Citrix, IBM, Microsoft, Sun and other companies. The standard has arisen to meet the growing demand from industry to create a portable format for virtual machines that is vendor and platform independent.  The goal of OVF is to provide a standard format that can robustly and efficiently deliver software solutions inside a set of virtual machines. The OVF format needs to be extensible and flexible enough to be able to serve the needs of describing a simple single VM image as well as describing the next-generation, dynamic applications for the cloud. In this blog entry, we will discuss the basic file structure of the OVF package, as well as many of the extensibility features built into the format.

On the configuration side, OVF 1.0 standardizes the basic settings for a hypervisor, including virtual hardware, disks, networks and resource allocation. An OVF package can also carry additional meta-data, including:

  • Product information that describes the software installed in the virtual machines, this covers both operating systems and application level components
  • End-user-license agreements (EULA)
  • Self-configuration parameters and deployment options for customization at deployment time

Ok, let's dive into the technical details to see how all this is achieved. We will cover three main areas: File layout and integrity, disk formats and compression, and the OVF descriptor and extensibility.

File Layout and Integrity

An OVF package consists of an OVF descriptor, a set of virtual disks, a set of localization bundles, a manifest, and a certificate (some of these are optional). For example:

MyPackage.ovf
MyPackage.mf
MyPackage.cert
MyPackage.vmdk
MyPackage.strings

The OVF descriptor (.ovf) is the main document of an OVF package. It contains all meta-data for the OVF package and has links to external files, such as virtual disks. We will get back into how this is structured in a moment.

The manifest (.mf) and certificate (.cert) files are optional and are used for integrity and authenticity checks. The manifest file contains the SHA1 digest of all files in the package (except for the .mf and .cert files themselves), and the certificate file contains a signed digest for the manifest file and an X.509 certificate. If present, they must be in the same directory as the OVF descriptor and have the same base name. We will skip string bundles for localization for now, that is a topic for a future blog entry.

An OVF package can be distributed as a set of discrete files as shown as above. For example, they can be uploaded to a web server and the URL to the OVF descriptor (.ovf) can be emailed to the recipients. However, often it is convenient to distribute a single file. The OVF specification defines a standard archive called an Open Virtualization Format Archive (.ova) exactly for this. This format is a "tarball" of the individual files that makes up the OVF package (with certain restrictions). For example:

MyPackage.ova

If you get an OVA package, try running tar tf <name>.ova and you can see the content. You can also create OVAs using the tar tool (or the OVF Tool or vSphere Client, of course). However, keep in mind that the OVA format is not simply a tar. It places certain restrictions on the ordering and naming of files. In particular, the OVF descriptor (.ovf) must be the first file and the files must be listed in the tar archive in the same order as they are listed in the OVF descriptor (see the OVF specification for all the details). These rules ensure that OVA archives are easy to stream - a tool or hypervisor does not need to download an entire OVA first and then unpack it.

Disk Formats and Compression

An OVF package is intended for distribution, so compression is the next big issue after ensuring package integrity. The granularity of compression is per file and works for both OVA archives and multi-files (.ovf) archives.  In other words, you don't need to create an OVA to get file compression.

In the OVF descriptor, you can specify compression options on each external reference, such as a string bundle or an ISO file (using the compression attribute in the Reference section). The most important part to compress is the virtual disk files. All OVF packages generated by VMware products utilize a variant of the VMDK format that is especially designed for distribution and already compressed; this is called the stream-optimized format. In this format, the content of a disk is stored, by default, in 64KB compressed chunks and only non-zero chunks are included. The format is designed so it is efficient to both consume and generate on the fly, as well as being able to provide good feedback on progress while being downloaded.  

Note that the VMDK format used for a deployed VM can and and typically will be different from the stream-optimized format. For instance, when deploying on ESX, the flat VMDK format is often used. The conversion from the stream-optimized format (used in the OVF package) to the flat VMDK format (used at runtime) is seamlessly done as part of the import step and is performed on the fly as part of the download process.

OVF Descriptor and Extensibility

Having covered the basic file level infrastructure, compression and integrity checking, let's take a look at the OVF descriptor (.ovf). The OVF descriptor is an XML document that stores or links to all information about the software contained in the OVF package. The purpose of the OVF descriptor itself is to provide the basic structure for embedding and discovering application meta-data. The outer most tag in the OVF descriptor is named Envelope, since the OVF descriptor is, in fact, modeled after an envelope where a variety of notes can be put into.  The following outlines the structure of an OVF descriptor:

<Envelope xmlns="http://schemas.dmtf.org/ovf/envelope/1" ... >
  <References>
     <File ovf:href="MyPackage.vmdk" ovf:id="disk1" ovf:size="68608"/>
  </References>
   ... List of OVF sections ...
   ... A Virtual System or VirtualSystemCollection ...
   ... String bundle references ...
</Envelope>

Each OVF descriptor has a required section, called References, that lists all file in the package  (except for the descriptor itself, the manifest and the certificate files). In this way a tool can always check an OVF package for integrity or copy it, without understand the content of each file or the meta-data sections that has included the external file references.

The rest of the OVF descriptor is made up of two parts: Entities and OVF sections. An entity is either a VirtualSystem or a VirtualSystemCollection, describing a single virtual machine or a container of multiple virtual machines, respectively. The syntax for describing a virtual machine is simply:

<VirtualSystem ovf:id="myVm">
     <Info> User-friendly description of the purpose of
            this entity (e.g., it describes a VM) </Info>
     <Name> Display name of the VM </Name>
     ... OVF Sections ...
</VirtualSystem>

An OVF package can also contain more than one virtual machine. This is done using the VirtualSystemCollection element:

<VirtualSystemCollection ovf:id="myVapp">
     <Info> User-friendly description of the purpose of the entity
            (e.g., it describes a VM) </Info>
     <Name> Display name of the vApp </Name>
    ... OVF Sections ...
    ... child entities - either VirtualSystem or VirtualSystemCollection ...
</VirtualSystemCollection>

The real meat of a VirtualSystem or VirtualSystemCollection is in the OVF sections. An OVF section is an XML fragment that contains the data for a specific functionality or aspect, such as virtual hardware requirements, operating system type, or maybe a dependency on an external system.  The general format of an OVF section is as follows:

<myns:MyOvfSection  ovf:required="true or false">
       <Info>A description of the purpose of the section</Info>
       ... section specific content ...
</myns:MyOvfSection>

The fully-qualified element tag (myns:MyOvfSection in the above example) uniquely identifies the OVF section.  The ovf:required and <Info> elements are part of the extensible design for OVF. OVF is designed to handle both forwards- and backwards compatibility. An OVF-compliant product cannot be expected to know about all possible sections, since OVF sections can be created by third-parties or defined in newer versions of the OVF specification. The ovf:required  attribute tells whether a tool can safely ignore the section, or whether it must fail if it does not understand the section. The <Info> provides a user-friendly description of the section that can be shown to the user in either case. Note that ovf:required defaults to true, so commonly it is left out.

An example of a custom third-party section in an OVF descriptor could be:

<myns:IncidentTrackingSection ovf:required="false">
   <Info>Useful info for incident tracking purposes</Info>
   <BuildSystem>Acme Corporation Official Build System</BuildSystem>
   <BuildNumber>102876</BuildNumber>
   <BuildDate>07-31-2009</BuildDate>
</myns:IncidentTrackingSection>

The OVF 1.0 specification defines 10 core sections that cover the basic vApp information:

  • DiskSection: Information about all virtual disks in the OVF package
  • NetworkSection: Defines the network topology inside an OVF package
  • ResourceAllocationSection: Resource settings for a VirtualSystemCollection
  • AnnotationSection: A description or annotation on an entity
  • ProductSection: Information about the software installed in the guest, including  properties for self-configuration
  • EulaSection: An end-user license agreement
  • StartupSection: Defines the start-up and shutdown order for the children in a VirtualSystemCollection
  • DeploymentOptionSection: Defines a set of pre-defined configurations that can be selected at deployment time
  • OperatingSystemSection: Specifies the operating system installed in a guest
  • InstallSection: Specifies whether a guest OS needs to be booted once (or more) before the application is fully deployed

These sections are described in the OVF Specification. See DSP0243 for the details, as well as the accompanying XML Schemas DSP8023 and DSP8027.  The OVF whitepaper DSP2017 also includes many examples on how these are used. The set of OVF sections are expected to grow significantly as the OVF format gains traction. 

We have only dived a little into the OVF descriptor, focusing on the file layout and the overall structure. We have covered the basic features around distribution support and extensibility. In future blog posts, we will dive into specific areas, such as localization, virtual hardware description, delta-disk compression, custom OVF sections and extensibility, multi-VM encodings, and much more.  So until next time, keep deploying those vApps!


Posted by Rene W. Schmidt on August 10, 2009 | Permalink | Comments (0) | TrackBack (0) |

07/14/2009

Self-Configuration and the OVF Environment

The OVF support in vSphere is not just about deployment and packaging. There is also a runtime part, namely the OVF environment. In this post, we will show a simple example on how the OVF environment can be used, by creating a VM template where you can customize its IP address. Consider the case where you deploy a VM from an OVF package or VM Template. We often want it to be customized just a little bit. A typical customization is IP settings (IP address, netmask, gateway, DNS, etc.), but also application-level settings, such as the location of the SNMP servers, admin logins, etc. Wouldn't it be nice to have a flexible, extensible, open-ended mechanism for doing this? This is exactly what the OVF environment is for. One way to think of the OVF environment is that the OVF environment is to a VM what the UNIX environment is to a UNIX process. So what does this mean? Well, the UNIX environment is a pretty simple mechanism that stores key/value pairs. These key/value pairs are typically used to tell a process or shell script about the particular environment it is being executed in. The UNIX environment has solved this problem nicely for processes, and the OVF team has been inspired by that feature and created the OVF properties and OVF environment and made it part of the Open Virtualization Format 1.0 specification.

OVF properties are typically configured by the deployer of a vApp, but can also be provided dynamically by vCenter. An example of a dynamic property is the vCenter server IP address. We will look into dynamic properties in a later post. To illustrate how to use the OVF environment, let us create a self-configuring VM template that configures its IP settings at startup. This is done by using OVF properties to convey information from vCenter to the VM, and installing a boot script in the VM that uses the information to (self) configure the guest software. Assuming that we have already installed the operating system into the VM, we need to perform the following steps: 

  • Define OVF properties on the VM.
  • Setup the OVF environment transport to carry the settings into the virtual machine.
  • Write some glue code to access and apply the information inside the VM.   

In the video below, we show how this is done using the vSphere client. In our example, we will create properties needed for IP configuration: IP, netmask, gateway, preferred DNS server, and alternate DNS server. When vCenter powers on a VM in a vApp, or a VM with vApp settings enabled, it creates an XML document called the OVF environment. The OVF environment contains all properties and their values. It is made available to the guest, allowing the guest to apply the properties to its configuration. When the VM is powered on, the OVF environment can also be viewed in the vSphere Client. The OVF environment can be transported to the guest in two ways:

  • As a CD-ROM containing the XML document. The CD-ROM is mounted on the guests CD-ROM drive.
  • Through VMware Tools. The variable guestinfo.ovfEnv contains the XML document.

In our example, the OVF environment will look like this:

<Environment oe:id="" xmlns="http://schemas.dmtf.org/ovf/environment/1" xmlns:oe="http://schemas.dmtf.org/ovf/environment/1" xmlns:ve="http://www.vmware.com/schema/ovfenv" xmlns:xml="http://www.w3.org/XML/1998/namespace" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <PlatformSection> <Kind>VMware ESX</Kind> <Version>4.0.0</Version> <Vendor>VMware, Inc.</Vendor> <Locale>en</Locale> </PlatformSection> <ve:EthernetAdapterSection> <ve:Adapter ve:mac="00:50:56:a0:3f:71" ve:network="VM Network"/> </ve:EthernetAdapterSection> <PropertySection> <Property oe:key="dns1" oe:value="10.20.60.1"/> <Property oe:key="dns2" oe:value="10.20.60.2"/> <Property oe:key="gateway" oe:value="10.20.63.253"/> <Property oe:key="ip" oe:value="10.20.63.219"/> <Property oe:key="netmask" oe:value="255.255.254.0"/> </PropertySection> </Environment>

The OVF environment must be read by some "glue" code in the guest - typically upon power-on. It is the responsibility of this code to configure the guest. In our example, we wrote a small Python script. It reads the property values from the XML file on the CD-ROM and configures the IP stack in the guest using netsh.exe: setupIp.py This script is called from a Windows scheduled task that is executed upon system startup. For Linux, a similar script can be added to /etc/init.d (depending on the Linux distribution). To access the OVF environment using VMware Tools instead of using the CD-ROM, we could execute the command: Windows: "C:\Program Files\VMware\VMware Tools\VMwareService.exe" -cmd "info-get guestinfo.ovfEnv"

Linux: vmware-guestd --cmd 'info-get guestinfo.ovfEnv'

In contrast to the normal VC template customization, there is a bit of extra setup required by the author of the template when using the OVF environment. The author is responsible for installing (and possibly writing) the glue code in the guest. However, you get complete freedom in which parameters you can customize, OS independence, and the deployment step is faster and more light-weight. In our example, we require the user to type in a fixed IP address and have detailed knowledge of various networking parameters. To improve this, we can use dynamic OVF properties to refer to vCenter's IP pools. We will look into this in a future blog post and also how you can share a self-customizing VM template using an OVF package.



Posted by Martin Amdisen on July 14, 2009 | Permalink | Comments (6) | TrackBack (0) |

07/07/2009

VMware Studio 2.0 Beta Unleashed

VMware Studio 2.0 is in public Beta and it comes with exciting feature set. While 1.0 release was focused on ISVs building virtual appliances, VMware Studio 2.0 has a broader scope. 

For developers, we introduce VMware Studio 2.0 Eclipse Plugin to provide an integrated dev/test environment using VMs. If you are the architect/lead developer and want to standardize development stack for your team, just create a build profile defining the components of your dev environment and distribute it or the resulting VM within your team. It is definitely better than creating a wiki with guidelines to setup development environments J 

If your application cannot be contained in a single virtual machine, you need a vApp! Simply put, vApps provide container semantics over a group of inter-related VMs. Take any n-tier application: CRM, ERP as an example. Such applications cannot be contained in a single virtual machine, generating a need to manage these virtual machines in the context of each other. By grouping them as a vApp, these VMs can be packaged, distributed, deployed and managed as one unit. vApp also provides a common OVF environment that is shared by contained VMs. And how do I create vApps? Use VMware Studio 2.0! With Studio 2.0 you can build vApps, author the OVF environment, specify startup/shutdown order of VMs and their resource requirements. You can distribute these VMs as a single OVF or OVA package. Learn more about deploying vApps in VC from Rene's blog on vApps 

Another area to touch upon is support for Windows. VMware Studio 2.0 enables building Windows 2003 and 2008 based VMs. This is our first release with Windows support. If you are an IT admin creating an “appliance” for internal distribution or developer standardizing windows based dev environment, give this feature a spin. 

Support for virtual appliances continues from Studio 1.0. Like physical appliances, virtual appliances can be treated as a black box. Think Linksys router, the only interface available to the users being a web console. Bringing the same concept into virtualization world, VMware Studio 1.0 embedded in-guest management framework – VAMI that is web accessible, into resulting appliances. It came with few in-guest services. With VMware Studio 2.0, VAMI is extensible. ISVs can write their custom in-guest services that plugs into this framework. The best way to create your own in-guest service is through “VMware Studio Linux Management Service” Wizard that is part of Eclipse Plugin for Studio. Check out Matt's Coffee talk webinar and demo for more details

Last but not the least, VMware Studio 2.0 enables fast builds. You can provide existing Studio-based VM as input, thereby avoiding building the VM from scratch each time. There is lot more to come in this area in future. We have just scratched the surface, so stay tuned.

To summarize VMware Studio 2.0 Features:

  • Build multi-VM vApps with rich metadata
  • Build Windows 2003 and 2008 based VMs
  • Support for building single VM and multi-VM Virtual Appliances  
  • Support for Ubuntu, RHEL, SLES, CentOS, 32 and 64 bit Linux VMs 
  • Eclipse Plugin
  • Extensible in-guest management framework
  • Accept existing Studio-built-VM as Input to build process
  • Better OS dependency resolution
  • Supports VMware Server 2.0, ESX 3.5 and 4.0, VC 2.5 and 4.0, Workstation 6.5.1 as provisioning engines along with VMware Server 1.0.x from Studio 1.0 
  • CLI interface for automation alongwith intuitive web interface

If this sounds exciting, download VMware Studio 2.0 Beta virtual appliance and deploy it in your favorite environment - ESX, WS or VMware Server. Ah and and before leaving, I have to mention this. We have put VMware Studio to a great use within VMware. VMware Studio is integrated it with our official build system and source control system to generate nightly builds of virtual appliances for various products, including VMware Studio 2.0! Talk about eating your own dog food..

Technorati Tags:


Posted by Komal Mangtani on July 7, 2009 in Web/Tech | Permalink | Comments (0) | TrackBack (0) |

07/01/2009

Command-line OVF Deployments

In the previous blog post, I showed how to deploy a multi-tiered OVF package using the Deploy OVF Template Wizard in the vSphere Client. The interactive workflow provided by the vSphere Client is hard to beat in terms of ease of use and simplicity. However, the graphical user interface also has it drawbacks. In particular, it can be fairly cumbersome to use if you have a large set of similar OVFs that needs to be deployed, or you want to automate a deployment. In those cases, a command-line utility is often preferred. Well, we have exactly the solution for that: OVF Tool 1.0.

OVF Tool provides a slew of different features, such as converting between OVF and .vmx formats and import/export of OVF 1.0 to vSphere 4.0, VirtualCenter 2.5/ESX 3.5 and earlier releases.

In the following I will show how to deploy the SugarCRM solution from the command-line and get exactly the same result as I got from the vSphere Client last week.

First, I probe the SugarCRM.ova to figure out what it contains and particular what parameters can be customized.

f:\>ovftool --hideEula http://aar-ovfrepo/ovf/SugarCRM.ova
Opening OVA source: http://aar-ovfrepo/ovf/SugarCRM.ova
OVF version:   1.0
Name:          SugarCRM
Version:       4.5.1e
Full Version:  4.5.1e-build 131
Vendor:        SugarCRM Inc
Product URL:   http://www.sugarcrm.com/crm/products/crm-products.html
Vendor URL:    http://www.sugarcrm.com/crm/

Annotation:  The sweet way to manage customer relationships.

End-user License Agreements:
  Present:     Yes (1)

Download Size:   764.49 MB

Deployment Sizes:
  Flat disks:     20.00 GB
  Sparse disks:    1.55 GB

Networks:
  Name:        Network
  Description: The network that the SugarCRM application will be available on

Virtual Hardware:
  Family:       vmx-04
  Disk Types:   SCSI-lsilogic

Properties:
  Key:         emailAdmin
  Category:    Application
  Label:       Administrator Email Address
  Type:        string
  Description: Enter email address for administrator. This is displayed on the help page.

  Key:         theme
  Category:    Application
  Label:       Theme
  Type:        string["Sugar", "RipCurl", "Retro", "Paradise", "Love",
               "Sunset"]
  Description: Select the default color/graphic scheme

  Key:         concurrentSessions
  Category:    Performance
  Label:       Concurrent Sessions
  Type:        int(10..1000)
  Description: The maximum allowed concurrent sessions.

  Key:         dbIp
  Category:    Network
  Label:       Database instance IP address
  Type:        ip:Network
  Description: IP address for the database instance (in dot-notation).

  Key:         webIp
  Category:    Network
  Label:       SugarCRM IP Address
  Type:        ip:Network
  Description: IP address on the SugarCRM application server. The service is made accessible at this IP address.

Deployment Options:
  Id:          small
  Label:       Evaluation
  Description: Use this configuration for evaluation purposes only. The number of CPUs required and amount of memory used is minimized, making it possible to run the system on a desktop system.

  Id:          medium
  Label:       Production
  Description: Standard settings for a typical product environment. This deployment option is suitable for a SMB production environment with less than 500 users.


  Id:          large
  Label:       Enterprise
  Description: Settings for large enterprise production environments. This deployment option is suitable for a large enterprise production environment with more than 500 users.

IP Allocation Policy:
  Schemes:     ovfenv dhcp
  Protocols:   IPv4

Completed successfully

By examining this output, I can gather the same information as was shown in the vSphere Client, such as product information, download sizes, end-user license agreements, deployment options, properties that can be customized, and supported IP policy schemes.

The next step is to deploy this to my vSphere 4.0 server. OVF Tool provides a handy pseudo-interactive mode for probing the vSphere inventory, so I do not have to open the vSphere client to look up the inventory organization or names of networks and datastores. To get started, I simply try and deploy it:

f:\>ovftool http://aar-ovfrepo/ovf/SugarCRM.ova vi://aar-dev-cluster-vc1
Opening OVA source: http://aar-ovfrepo/ovf/SugarCRM.ova
Please enter login information for target vi://aar-dev-cluster-vc1/
Username: VMWAREM\renes
Password: *********
Error: Found wrong kind of object (Folder)
Possible completions are:   
  aar-dev-datacenter/
  Jan's Test Datacenter/   

Ok, so the first completion is: vi://aar-dev-cluster-vc1/aar-dev-datacenter/. After a few more iterations, I get to this:

f:\>ovftool --acceptAllEulas http://aar-ovfrepo/ovf/SugarCRM.ova vi://VMWAREM%5Crenes@aar-dev-cluster-vc1/aar-dev-datacenter/host/Cluster/Resources/DemoPool
Opening OVA source: http://aar-ovfrepo/ovf/SugarCRM.ova
Please enter login information for target vi://aar-dev-cluster-vc1/
Username: VMWAREM\renes
Password: *********
Opening VI target: vi://VMWAREM\renes@aar-dev-cluster-vc1/aar-dev-datacenter/host/Cluster
Error: No target datastore specified
Datastores found on target:
  Cluster VMFS
  Storage1

It now provides completions for datastores. Simlilarly, OVF Tool will provide completions for networks (if there are multiple choices). The final command to deploy, customize, and power-on the SugarCRM OVF package is:

f:\>ovftool "--datastore=Cluster VMFS"
            "--network=VM Network"
            --acceptAllEulas
            -ipAllocationPolicy=transient
            --prop:emailAdmin=admin@vmware.com
            --prop:theme=Retro
            --powerOn
            http://aar-ovfrepo/ovf/SugarCRM.ova
            vi://VMWAREM%5Crenes@aar-dev-cluster-vc1/aar-dev-datacenter/host/Cluster/Resources/DemoPool
Opening OVA source: http://aar-ovfrepo/ovf/SugarCRM.ova
Please enter login information for target vi://aar-dev-cluster-vc1/
Username: VMWAREM\renes
Password: *********
Opening VI target: vi://VMWAREM\renes@aar-dev-cluster-vc1/aar-dev-datacenter/host/Cluster/Resources/DemoPool
Target: vi://aar-dev-cluster-vc1/aar-dev-datacenter/host/Cluster/Resources/DemoPool
Disk Transfer Completed
Powering on vApp: SugarCRM
Completed successfully

Voila! (Standard disclaimer: I inserted a few line breaks to make it more readable. How to escape spaces in parameters varies whether you are on Linux or Windows).

This was just a very quick preview of the features of OVF Tool. Consult the documentation for more examples, and download the tool today to try it out for yourself. Next time, I will dive into the internal structure of the SugarCRM OVF package.


Posted by Rene W. Schmidt on July 1, 2009 in Weblogs | Permalink | Comments (8) | TrackBack (0) |

Next »

About VMware vApp Blog

  • In this blog, we will dig into the details of OVF, vApps, and virtual appliances and how they can be put to practical use - both by IT administrators and virtual appliance authors. If you are wondering about the technical details and how to apply OVF in practice, this is a good place to learn more.

Subscribe

Resources

Recent Comments

Archives