Home > Blogs > VMware vFabric Blog


5 Scenarios and Best Practices for Running Multiple Instances of Tomcat or tc Server

With multiple Tomcat instances, each runs in its own JVM, with its own configuration, and can be started or stopped independently, while still running against the same core binary. There are a variety of reasons to do this in practice. For example:

  1. Simplify updates by separating instance specific data like web applications from the core Tomcat software.
  2. Maintain central control (and restricted permissions) on core Tomcat software, while allowing Tomcat instances to run as individual users without root permissions.
  3. Isolate web applications to a particular Tomcat instance for protection from faults in other applications.
  4. Permit application-specific performance monitoring (and usage billing) by having each application in its own Tomcat instance.
  5. Configure the Java Virtual Machine specifically for the needs of the application(s) running on that Tomcat instance.

Configuring Tomcat such that a single binary runtime directory supports multiple independent instances is a simple matter of creating the correct directory hierarchies and setting a couple of environment variables. vFabric tc Server automates these tasks, but uses the same underlying mechanism as Tomcat.  Given these basic facts, it’s easy to adopt a tc Server best practice for use with Tomcat.

Background

Learn More

Register for VMworld!
Click Here

Register for Session APP-CAP1676 – VMware vFabric tc Server Best Practices for Security, Stability and Sanity:
Click Here

Follow all vFabric updates at VMworld on Twitter:
Click Here

For a very long time, Tomcat has had the ability to define and run multiple instances from a single copy of a binary distribution, and this functionality is well documented in the file RUNNING.txt at the root of the Tomcat binary distribution file hierarchy. But in my experience, not all Tomcat users are aware of this capability or understand its advantages. On the other hand, users of VMware vFabric tc Server use this feature by default due to the structure of tc Server and the convenience scripts provided with it. Both Tomcat users and tc Server users can benefit from a related best practice I describe in my VMworld (Barcelona) presentation, APP-CAP1676, tc Server Best Practices for Security, Stability, and Sanity. In this article, I review the mechanics of the multiple instance functionality, note how the scripts provided with tc Server facilitate its use, and then suggest a simple, related best practice for security on a system hosting multiple instances (of either Tomcat or tc Server).

How it Works

As previously noted, the file RUNNING.txt is the authoritative document on how to configure Tomcat for multiple instances. Let me summarize. To run with multiple instances, you will have one directory containing the files that make up the Tomcat runtime as distributed by the Apache Software Foundation, and another directory for each individual instance. The path to the core runtime is stored in the environment variable CATALINA_HOME; the path to the active configuration (a.k.a. instance) is stored in the environment variable CATALINA_BASE. A typical CATALINA_BASE directory will contain subdirectories bin, conf, logs, webapps, work, and temp. Let’s run through the purpose of each of those directories. (Note that this document presumes Tomcat running on a Unix variant, but for Windows the techniques are analogous. For instance, substitute the suffix .bat for .sh.)

  • bin — Optional. When present, contains setenv.sh to set instance specific environment variables and the variant of tomcat-juli.jar you wish to use for that instance (depending on whether you want to use log4j).
  • conf — Server configuration files, including server.xml
  • lib — Libraries and classes, specific to that instance. Jar and class files here will take precedence over the same ones in CATALINA_HOME/lib.
  • logs — Log and output files.
  • webapps — Automatically loaded web applications.
  • work — Temporary working directories for web applications.
  • temp — Directory used by the JVM for temporary files.

The directories under CATALINA_BASE are specific to an instance, where the directories under CATALINA_HOME are common to all instances.

To start up a separate instance, you set CATALINA_HOME and CATALINA_BASE appropriately, along with other required environment variables, such as JAVA_HOME or JRE_HOME and then run $CATALINA_HOME/bin/startup.sh. CATALINA_HOME must be set; however, if CATALINA_BASE is not set, then the Tomcat script catalina.sh (invoked by startup.sh) sets it to the value of CATALINA_HOME.

Of course, when running multiple instances simultaneously on the same system, you have to make sure that certain configuration parameters, such as port numbers for Connectors, are unique.

One nice benefit of a separate CATALINA_BASE directory is being able to easily run Tomcat with different Java runtime arguments. You may want to run one instance with a 2GB heap, and another instance with a 256MB heap. This is accomplished by editing $CATALINA_BASE/bin/setenv.sh to set CATALINA_OPTS to the appropriate value,  -Xmx2G or -Xmx256M.

tc Server and Multiple Instances

Users of vFabric tc Server transparently use multiple instances through the use of the tcruntime-instance.sh and tcruntime-ctl.sh scripts. When you create an instance with the command

prompt% ./tcruntime-instance.sh create node1

the tcruntime-instance.sh script creates a directory named node1 containing subdirectories bin, conf, lib, logs, temp, webapps, and work. The tcruntime-ctl.sh script, used to start and stop the instance, sets CATALINA_HOME to the proper runtime directory (which for tc Server could be Tomcat 6 or Tomcat 7), sets CATALINA_BASE to the directory created by tcruntime-instance.sh, and then starts up (or shuts down) the application server.

There are a number of significant advantages to having separate CATALINA_HOME and CATALINA_BASE directories. For one, it makes upgrading to a new release clean and easy. All you need to do is unpack the Tomcat distribution in its own directory and then set CATALINA_HOME to this new directory before running startup.sh. (Note that this works only for point releases; the differences between major releases of Tomcat, such as 6 and 7, are too significant for this to work in most cases.) Similarly, you can simultaneously run multiple instances that use different minor versions of Tomcat.

Best Practice: Non-Root Ownership of CATALINA_BASE

This division of core runtime files from instance-specific configuration and web application files allows Tomcat users to follow a convention recommended as a best practice for tc Server: creating and running a Tomcat instance as a non-root user.

For tc Server, the default behavior of tcruntime-instance.sh is to create the CATALINA_BASE directory at the root of the tc Server installation. By default, if tc Server was installed with yum/rpm (and by convention otherwise) that directory is /opt/vmware/vfabric-tc-server-standard/.  This directory is owned by root, so unless an administrator modifies the owner and/or permissions on the directory, non-root users cannot create subdirectories therein. Which means that non-root users cannot use tcruntime-instance.sh to create an instance without specifying an alternative instance directory with the -i option. The parallel situation for Tomcat users is where the Tomcat distribution has been installed in a directory owned by root (let’s assume this directory is /usr/local/apache-tomcat-7.0.30). In a well-controlled environment, It makes sense for root (or another privileged user) to have exclusive write permission to the Tomcat install hierarchy. You don’t want random users to be able to modify the runtime used by every Tomcat instance. A consequence of this is that non-privileged users cannot run Tomcat without setting CATALINA_BASE to a suitable directory for which they have write permission. This is because a running Tomcat needs to have permission to create directories and files (e.g. exploded war files in the webapps directory, log files in the logs directory) under CATALINA_BASE. If CATALINA_BASE is not set, it defaults to CATALINA_HOME and since this directory has restricted access, non-privileged users cannot create the files and directories required for Tomcat to run.

Obviously, it is not good to require (or even allow) root to run Tomcat. As mentioned previously, the solution for tc Server users is to specify an alternative location for CATALINA_BASE with the -i option to tcruntime-instance.sh. The rpm install creates a directory specifically for this purpose, /var/opt/vmware/vfabric-tc-server-standard, owned by the user tcserver, so a proper use of the command would be

prompt% ./tcruntime-instance.sh create node1 -i /var/opt/vmware/vfabric-tc-server-standard

This creates the directory /var/opt/vmware/vfabric-tc-server-standard/node1, which will be CATALINA_BASE for this instance.

For Tomcat, one would designate a directory, such as /var/local/tomcat, as the standard location for all CATALINA_BASE directories. This directory’s ownership, group, and permissions could be configured to allow members of a designated group, say tomcat, to create subdirectories therein. By convention, any CATALINA_BASE directory would be located there, and Tomcat would be invoked with this sequence of commands:

prompt% export CATALINA_HOME=/usr/local/apache-tomcat-7.0.30
# node1 configured with proper subdirectories: bin, conf, logs, temp, webapps, work
prompt% export CATALINA_BASE=/var/local/tomcat/node1
prompt% $CATALINA_HOME/bin/startup.sh

Now that you’ve got a good overview, you can explore using multiple instances to more cleanly separate the core Tomcat binaries from the applications that will run in the Tomcat container.

>> For more information on VMware vFabric tc Server, please download a trial, read the documentation, or check out the resource library.

About the Author: Channing Benson, currently a Java and vFabric Solutions Architect at VMware, has over 20 years of experience in the computer industry. Throughout his tenure he has helped customers adopt and master developing technologies, starting with  X11/Motif on Unix-based workstations, cross-Unix porting and C language optimization, and Linux development and performance tuning. While working at HP, he contributed to the FOSSology Open Source project. Channing has been a Java programmer/architect since its genesis in 1995, working primarily to help customers and independent software vendors optimize their programs for the latest available platforms, from EJBs and J2EE to Spring on lightweight containers like Tomcat and tc Server, and now cloud hosts like Cloud Foundry and high performance data grids like GemFire.

6 thoughts on “5 Scenarios and Best Practices for Running Multiple Instances of Tomcat or tc Server

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>