Home > Blogs > VMware Hands-On Lab (HOL) Blog > Monthly Archives: February 2017

Monthly Archives: February 2017

HOL Three-Tier Application, Part 5 – Use Cases

If you have been following along in this series, first of all, thank you! Here is a summary of our work so far:

Next, you should have a basic three-tier application created:


A Simple Three-Tier Application

I tried to use simple components to make it usable in either a home lab or a nested environment, so they should perform exceedingly well in a real environment.

Virtual Machine Profile

The component Photon OS machines boot in a few seconds, even in our nested environment, and their profiles are fairly conservative:

  • 1 vCPU
  • 2 GB RAM
  • 15.625 GB disk

Once configured as indicated in this series, these VMs will export as OVAs that are around 300 MB each, making them reasonably portable.

The storage consumed after thin-provisioned deployment is less than 650 MB for each virtual machine. At runtime, each consumes an additional 2 GB for the swapfile. During boot, in my environment, each VM’s CPU usage is a little over 600 MHz and the active RAM reports 125 MB, but those normalize quickly to nearly 0 MHz and 20 MB active RAM (+23 MB virtualization overhead). You may be able to reduce their RAM allocations, but I have not tried this.

So, what can I do with this thing?

It is nice to have tools, but without a reason to use them, they’re not that much fun. We use tools like this in our labs to demonstrate various functionality of our products and help our users understand how they work. Here are a few ideas, just to get you thinking:

vMotion, Storage vMotion, SRM Protection and Recovery

The virtual machines that you created can be used as a set, but the base Photon OS template also makes a great single VM for demonstrating vMotion or Site Recovery Manager (SRM) recovery in a lab environment. They are small, but they have some “big VM” characteristics:

  • The VMware Tools provide appropriate information up to vCenter
  • They respond properly to Guest OS restart and power off actions
  • Photon OS handles Guest Customization properly, so you can have the IP address changed during template deployment and SRM recovery.
  • You can ping and SSH into them
  • You can use them to generate load on your hosts and demonstrate Distributed Resource Scheduler (DRS) functionality


We use a previous version of this application in several of our NSX labs that debuted at VMworld 2016. For a good micro-segmentation use case, you can look at HOL-1703-USE-2 – VMware NSX: Distributed Firewall with Micro-Segmentation. The manual is available for download here, or you can take the lab here.

For a more complicated use case using a similar application to demonstrate SRM and NSX integration, look at HOL-1725-USE-2 – VMware NSX Multi-Site DR with SRM. For that lab, the manual is available here and the lab is here.

Each of the tiers must communicate with the others using specific ports

  • Client to Web = 443/tcp
  • Web to App = 8443/tcp
  • App to DB = 80/tcp

You can use this application to test firewall rules or other network restrictions that you are planning to implement. If a restriction breaks the application, you can determine where and why, then try again. If you want to change the port numbers to match your needs, you can do that as well. Keeping the application simple means that modifications should also be simple.

Load Balancing (Distribution)

The basic idea here is that you can create clones of the web-01a machine as many times as you like and pool them behind a load balancer. In your lab, if you have it, you may want to use NSX as a load balancer. If you want to do that, I suggest checking out Module 3 – Edge Services Gateway in the HOL-1703-SDC-1 – VMware NSX: Introduction and Feature Tour lab, which covers how to set that up. The manual is here and the lab is here.

If you want to use another vendor’s solution, feel free to do that as well. This application is REALLY simple. Some free load balancing solutions can be implemented using nginx or haproxy. Fortunately, we already know about nginx from the build of our web servers, so I will cover that later in this post. First, though, I want to cover a DNS round robin configuration since understanding that makes the nginx load balancing simpler for the lab.

Example 1 – Load Distribution via DNS Round Robin

If you don’t have the resources for another VM, you can implement simple load distribution via DNS round robin as long as you understand a few limitations:

  1. You must have access to change DNS for your lab environment.
  2. Using only DNS, you get load distribution but not really balancing; there is no awareness of the load on any particular node. Rather, you simply get the next one in the list.
  3. There is no awareness of the availability of any node in the pool. DNS simply provides the next address, whether it is responding or not.
  4. Connecting from a single client may not show balancing since optimizations in modern web browsers may keep existing sockets open.

In this first example, I have 3 web servers (web-01a, web-02a, web-03a) with IP addresses, 31, and 32. My SSL certificate contains the name webapp.corp.local and it is loaded onto each of the web servers. The picture looks something like this:


Create the VMs

To create web-02a and web-03a, I simply clone my web-01a VM then reset the hostnames and IP addresses of each clone to the new values:

  • web-02a –
  • web-03a –

Alternatively, I can make a template from the web-01a VM and deploy the copies using Guest Customization to reconfigure them. Just make sure to populate the /etc/hosts file on the customized machines since the process wipes out and rebuilds that file.

Configure DNS

The required DNS changes are not complicated. You basically assign the name webapp.corp.local to the IP addresses of your web servers and set the time-to-live (TTL) to a low, non-zero value.

Using PowerShell against my lab DNS server called controlcenter.corp.local that manages the corp.local zone, I add DNS records with a 1 second TTL, associating all of the web server IP addresses to the name webapp.corp.local:

$ttl = New-TimeSpan -Seconds 1

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webapp' -IPv4Address '' -TimeToLive $ttl

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webapp' -IPv4Address '' -TimeToLive $ttl

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webapp' -IPv4Address '' -TimeToLive $ttl

If you use a BIND DNS server, just create multiple A records pointing to the same name. BIND 4.9 or higher will automatically rotate through the records. In my case, I have a Windows 2012 DNS server, and it cycles through the addresses when the webapp.corp.local name is requested.

Testing the Rotation

Here is a simple example of what this looks like from an ESXi host in my lab. A simple ping test shows the rotation occurring as intended:

[root@esx-03a:~] ping -c 1 webapp.corp.local
PING webapp.corp.local ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=1.105 ms

--- webapp.corp.local ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.105/1.105/1.105 ms

[root@esx-03a:~] ping -c 1 webapp.corp.local
PING webapp.corp.local ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=1.142 ms

--- webapp.corp.local ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.142/1.142/1.142 ms

[root@esx-03a:~] ping -c 1 webapp.corp.local
PING webapp.corp.local ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=1.083 ms

--- webapp.corp.local ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.083/1.083/1.083 ms

Accessing the Application

Use the https://webapp.corp.local/cgi-bin/app.py URL from your web browser to access the application. Within the three-tier application, the script on the app server displays which web server made the call to the application.


The script will show the IP address of the calling web server unless it knows the name you would like it to display instead. You provide a mapping of the IPs to the names you would like displayed at the top of the app.py script on the app server:

webservers = {

Simply follow the syntax and replace or add the values which are appropriate for your environment.

A Challenge Showing Load Distribution from a Single Host

Hmm… while the ping test shows that DNS is doing what we want, clicking the Refresh button in your web browser may not be switching to a different web server as you expect.

A refresh does not necessarily trigger a new connection and DNS lookup, even if the TTL has expired. Modern web browsers implement optimizations that will keep an existing connection open because odds are good that you will want to request more data from the same site. If a connection is already open, the browser will continue to use that, even if the DNS TTL has expired. This means that you will not connect to a different web server.

You can wait for the idle sockets to time out or force the sockets closed and clear the web browser’s internal DNS cache before refreshing the web page, but that is not really convenient to do every time you want to demonstrate the distribution functionality. If you want to be able to click Refresh and immediately see that you have connected to a different web server in the pool, you can use NSX or a third-party load balancer. If you want to use the tools that we have currently available, the next example works around this issue.

Example 2 – Implementing a (Really) Basic Load Balancer

Making a small change to the nginx configuration on one of the web server machines and adjusting DNS can provide a simple demonstration load balancer for your lab. This requires a slight deviation from our current architecture to inject the load balancer VM in front of the web server pool:


Three-Tier Application with Load Balancer

Note that there are better, more feature-rich ways to do this, but we are going for quick and simple in the lab.

Create the Load Balancer

Create the load balancer VM. You can deploy a new one from a Photon OS base template and go through the configuration from there, but conveniently, the difference between the load balancer configuration and that of our web servers is just one line!

So, make a copy of the web-01a VM and update its address and hostname:

  • lb-01a –

Change the nginx Configuration

On the lb-01a VM, edit the /etc/nginx/nginx.conf file

# vi +130 /etc/nginx/nginx.conf

Change line 130 from

proxy_pass https://app-01a.corp.local:8443/;


proxy_pass https://webpool.corp.local/;

This will allow us to leverage DNS round-robin to rotate through the list of web servers and distribute the load. Nginx has advanced configurations to handle load balancing, but this will get the job done for a lab or demonstration. Terminating SSL on the load balancer while using plain HTTP on the web servers allows a lot more flexibility, but the configuration changes are beyond the scope of what I want to do here.

Restart nginx

# systemctl restart nginx

Adjust DNS

Finally, adjust DNS to move the webapp.corp.local name to point at the load balancer and put the web servers into webpool.corp.local instead.

If you are using Windows DNS, you can use PowerShell. For BIND, edit and create the records as needed.

  1. Remove the existing webapp.corp.local pool by deleting all of the A records that point to the individual web servers:
$rec = Get-DnsServerResourceRecord -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -Name 'webapp' -RRType A
if( $rec ) { 
  $rec | % { Remove-DnsServerResourceRecord -InputObject $_ -ZoneName 'corp.local' -Force }

2. Create a new webapp.corp.local A record that points to the lb-01a machine:

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webapp' -IPv4Address ''

3. Create the new webpool.corp.local that contains the individual web servers:

$ttl = New-TimeSpan -Seconds 1

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webpool' -IPv4Address '' -TimeToLive $ttl

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webpool' -IPv4Address '' -TimeToLive $ttl

Add-DnsServerResourceRecordA -ComputerName 'controlcenter.corp.local' -ZoneName 'corp.local' -name 'webpool' -IPv4Address '' -TimeToLive $ttl

Access the Application

Now, point your web browser to the https://webapp.corp.local/cgi-bin/app.py URL. Each time you click Refresh in your web browser or enter a new search string in the Name Filter box and click the Apply button, the data refresh and the Accessed via: line should update with a different web server from the pool:


Rotating through web servers in the pool

Because the web browser’s connection is to the load balancer VM, which controls which web server receives the request, we eliminate the issue experienced when using only DNS round robin. This very basic implementation does not handle failed servers in the pool and is not something that would be used in production, but, hey, this is a lab!

It is possible to extend this idea to put a load balancer in front of a pool of application servers as well: replace line 130 in each web server’s /etc/nginx/nginx.conf file with the URL of an app server pool instead of pointing them directly at the app-01a VM.

That’s a Wrap!

That concludes the series on building a minimal three-tier application. I am hopeful that you have found this interesting and can use these tools in your own environment.

Thank you for reading!

HOL Three-Tier Application, Part 4 – Web Server

This is the fourth post in the series about building a three-tier application for demonstration, lab, and education purposes. If you have been following along, you have created the base Photon template as well as simple database and application servers.

This post covers the final layer of our stack, the web presentation tier. I have said it before, but the configuration of the web server here is really simple, and if you have made it this far, you’re golden.

The Web Server (web-01a)

All of the hosts in this application run “web server” software, but this one has the web server designation because it is the one that the user directly accesses. The entire back end could be replaced with real application middleware and an RDBMs, but the user expects this one to present an SSL-encrypted web page on port 443. This time, I have chosen not to use Apache, and there is no need for Python. There is no CGI, and minimal configuration is required aside from issuing another certificate for the SSL. This tier is mostly interesting because it will support the virtual name of the application in addition to the real name(s) of your web server(s).

The red box in the following diagram highlights the component that we are building in this post.


Again, the first steps look quite a bit like the steps we performed for the application and database servers as we assign the personality to the template. I will again outline the steps here as a reminder. Details can be found in my post about the database server.

Let’s get started!

  1. Deploy a copy of the base Photon template you created by following the steps in my first post.
  2. Name it something that makes sense to you for your web server. I called mine web-01a
  3. Power it up and log in as the root user
  4. Change the hostname in /etc/hostname and in /etc/hosts
  5. Change the IP address in /etc/systemd/network/10-static-eth0.network
  6. Use a SSH client to access the machine as root (makes pasting possible)

Instead of installing Apache again, we are going to use nginx. You can do the same thing with Apache, but I wanted to try something a little more lightweight and the nginx configuration for this use case is really simple.

Install nginx to be used as a reverse proxy

The web server machine will function as a reverse proxy, sending user requests bound for port 443 on this server to the application server at https://app-01a.corp.local:8443

# tdnf install nginx

The nginx install is less than 6 MB and takes a few seconds:


Configure the reverse proxy

Edit the configuration file, /etc/nginx/nginx.conf

# vi +116 /etc/nginx/nginx.conf

by adding the following at the bottom of the file, at line 116, just before the closing “}” in the file.

   # HTTPS server
   server {
      listen 443;
      server_name webapp.corp.local;

      ssl on;
      ssl_certificate     /etc/nginx/ssl/webapp.pem;
      ssl_certificate_key /etc/nginx/ssl/webapp.key;

      ssl_session_cache shared:SSL:1m;
      ssl_session_timeout 2m;

      location / {
         proxy_pass https://app-01a.corp.local:8443/;
         proxy_set_header Host $host;
         proxy_redirect http:// https://;

Notice that we need an SSL certificate and a key to make this work. We have done this before, so let’s create those now.

Make the ssl directory and switch into it

Let’s just create the certificates in the place that the server expects them to be.

# mkdir -p /etc/nginx/ssl
# cd /etc/nginx/ssl

Build the configuration file

It is common for multiple web servers to be configured in a pool, behind a load balancer. I create the certificate here using a name, webapp.corp.local. This name can be assigned to the load balancer’s VIP. If there is only one web server, as in my example here, this name can also be an alias that resolves to the one web server. For simplicity, and possibly for other use cases, the certificate configuration we build here includes the names of three web servers: web-01a, web-02a and web-03a.

Create the file webapp.conf

# vi webapp.conf

with the following contents, modified as needed for your environment:

distinguished_name = req_distinguished_name
x509_extensions = v3_req
default_md = sha256
prompt = no
C = US
ST = California
L = Palo Alto
O = VMware
OU = Hands-on Labs
CN = webapp.corp.local
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
DNS.1 = webapp.corp.local
DNS.2 = web-01a.corp.local
DNS.3 = web-02a.corp.local
DNS.4 = web-03a.corp.local

Save and close the file.

Generate the key and certificate

Note that this is a long command and you may need to scroll to the right to get all of it. Ensure it ends with “webapp.conf”

# openssl req -x509 -nodes -days 1825 -newkey rsa:2048 -keyout webapp.key -out webapp.pem -config webapp.conf

(optional) Validate that the PEM file “looks right”

I put this command here for those who want to look at the certificate. It is a good command to know in case you have a certificate file and want to know what information it contains. That can help you match the certificate to the proper host without needing to install it and then find out it is not the right one.

# openssl x509 -in webapp.pem -noout -text

Start the nginx server and configure it to startup when the VM boots

# systemctl start nginx
# systemctl enable nginx


With the other components (db-01a, app-01a) online, reachable and tested, you can test the whole solution with curl from the console of the web server

# curl -k https://web-01a/cgi-bin/app.py

This should return the data from the database in HTML format by executing the script on the application server.

You can filter the results by appending a querystring. Try this one:

# curl -k https://web-01a/cgi-bin/app.py?querystring=science

That query should return a single entry with a name containing the word science. It may be difficult to read on the command line since it is HTML. These look nicer via a GUI web browser anyway, and you can modify the filter using the form at the top of the table:


That’s it!

You now have the components of a rudimentary three-tier web application that you can use in your lab. I hope this build has provided some useful tools for you. In the final post, I will use this set of VMs and cover an example of how to implement a pool of webservers in front of the application and database tier.

Thank you for reading!

Oh, just one more thing…

Notice the pretty green lock next to the URL in my web browser in the previous screen shot?

SSL Certificate Trust

In this application, we have a self-signed SSL certificate. It should be created with the name webapp.corp.local, or whatever you selected for your environment. To get rid of the web browser security warnings and have the shiny green lock show up, you need to do two things:

Configure DNS Records

The only record you need from the client side is one that points to webapp.corp.local.

If you have a Windows-based DNS server, you can create the records using PowerShell. The following 2 lines create a DNS host (A) record for web-01a.corp.local and then a DNS alias (CNAME) record for webapp.corp.local that points to it.

PS> Add-DnsServerResourceRecordA -ComputerName 'dns.corp.local' -ZoneName 'corp.local' -name 'db-01a' -IPv4Address '' -CreatePtr

PS> Add-DnsServerResourceRecordCName -ComputerName 'dns.corp.local' -ZoneName 'corp.local' -Name 'webapp' -HostNameAlias 'web-01a.corp.local.'

This configuration allows the virtual name webapp to be separate from the web-01a name and enables the addition of other web servers to a pool, followed by the reassignment of the webapp name to a load balancer IP.

If you don’t have Windows-based DNS, you can edit your /etc/hosts file on the client or add the DNS records to your nameserver using the procedures required for your environment.

Trust the Self-Signed Certificate

Once you have name resolution knocked out, you need to trust the certificate on your client. You can really trust the certificate, or you can sometimes create an exception in the web browser. Do whichever works for you and makes you happy. Without trust, this is what the connection looks like in Chrome:


In our labs, we download the web server’s certificate to the client machine and add it to the Windows Trusted Root Certification Authorities store or one of the subtrees within Keychain Access on MacOS. That will handle IE on Windows, and Chrome browsers on Windows and MacOS.

If you save the certificate file to your desktop in Windows and double-click it, the bold text pretty much sums up what you need to do.


There are a variety of ways to get this done, and there are some shortcuts, but the process has not changed in many years and this Microsoft Windows Blog article covers a process that works.

Firefox manages its own trust store, so you need to import it separately if you want to use that browser. Check the Mozilla Wiki for detailed instructions about how to do this. Note that newer versions of Firefox have implemented more strict checking. Basically, they refuse to accept a “leaf” certificate that is specified as a Certificate Authority certificate (why is your web server using the CA certificate??) and will not allow a non-CA certificate to be added to its trusted root CA certificate store. Getting this to look nice requires additional hoops that are beyond the scope of this article. We have a Microsoft CA implemented in our labs and generally issue certificates from there. Since that CA is trusted by all clients within the environment, there is no issue.

Thank you again for reading!

You can finish the series with the last post: Use Cases.