With multiple Tomcat instances, each runs in its own JVM, with its own configuration, and can be started or stopped independently, while still running against the same core binary. There are a variety of reasons to do this in practice. For example:
- Simplify updates by separating instance specific data like web applications from the core Tomcat software.
- Maintain central control (and restricted permissions) on core Tomcat software, while allowing Tomcat instances to run as individual users without root permissions.
- Isolate web applications to a particular Tomcat instance for protection from faults in other applications.
- Permit application-specific performance monitoring (and usage billing) by having each application in its own Tomcat instance.
- Configure the Java Virtual Machine specifically for the needs of the application(s) running on that Tomcat instance.
Configuring Tomcat such that a single binary runtime directory supports multiple independent instances is a simple matter of creating the correct directory hierarchies and setting a couple of environment variables. vFabric tc Server automates these tasks, but uses the same underlying mechanism as Tomcat. Given these basic facts, it’s easy to adopt a tc Server best practice for use with Tomcat.
For a very long time, Tomcat has had the ability to define and run multiple instances from a single copy of a binary distribution, and this functionality is well documented in the file RUNNING.txt at the root of the Tomcat binary distribution file hierarchy. But in my experience, not all Tomcat users are aware of this capability or understand its advantages. On the other hand, users of VMware vFabric tc Server use this feature by default due to the structure of tc Server and the convenience scripts provided with it. Both Tomcat users and tc Server users can benefit from a related best practice I describe in my VMworld (Barcelona) presentation, APP-CAP1676, tc Server Best Practices for Security, Stability, and Sanity. In this article, I review the mechanics of the multiple instance functionality, note how the scripts provided with tc Server facilitate its use, and then suggest a simple, related best practice for security on a system hosting multiple instances (of either Tomcat or tc Server).
How it Works
As previously noted, the file RUNNING.txt is the authoritative document on how to configure Tomcat for multiple instances. Let me summarize. To run with multiple instances, you will have one directory containing the files that make up the Tomcat runtime as distributed by the Apache Software Foundation, and another directory for each individual instance. The path to the core runtime is stored in the environment variable
CATALINA_HOME; the path to the active configuration (a.k.a. instance) is stored in the environment variable
CATALINA_BASE. A typical
CATALINA_BASE directory will contain subdirectories bin, conf, logs, webapps, work, and temp. Let’s run through the purpose of each of those directories. (Note that this document presumes Tomcat running on a Unix variant, but for Windows the techniques are analogous. For instance, substitute the suffix .bat for .sh.)
- bin -- Optional. When present, contains setenv.sh to set instance specific environment variables and the variant of tomcat-juli.jar you wish to use for that instance (depending on whether you want to use log4j).
- conf -- Server configuration files, including server.xml
- lib -- Libraries and classes, specific to that instance. Jar and class files here will take precedence over the same ones in CATALINA_HOME/lib.
- logs -- Log and output files.
- webapps -- Automatically loaded web applications.
- work -- Temporary working directories for web applications.
- temp -- Directory used by the JVM for temporary files.
The directories under
CATALINA_BASE are specific to an instance, where the directories under
CATALINA_HOME are common to all instances.
To start up a separate instance, you set
CATALINA_BASE appropriately, along with other required environment variables, such as
JRE_HOME and then run
$CATALINA_HOME/bin/startup.sh. CATALINA_HOME must be set; however, if
CATALINA_BASE is not set, then the Tomcat script catalina.sh (invoked by startup.sh) sets it to the value of
Of course, when running multiple instances simultaneously on the same system, you have to make sure that certain configuration parameters, such as port numbers for Connectors, are unique.
One nice benefit of a separate
CATALINA_BASE directory is being able to easily run Tomcat with different Java runtime arguments. You may want to run one instance with a 2GB heap, and another instance with a 256MB heap. This is accomplished by editing
$CATALINA_BASE/bin/setenv.sh to set
CATALINA_OPTS to the appropriate value, -Xmx2G or -Xmx256M.
tc Server and Multiple Instances
Users of vFabric tc Server transparently use multiple instances through the use of the tcruntime-instance.sh and tcruntime-ctl.sh scripts. When you create an instance with the command
prompt% ./tcruntime-instance.sh create node1
the tcruntime-instance.sh script creates a directory named node1 containing subdirectories bin, conf, lib, logs, temp, webapps, and work. The tcruntime-ctl.sh script, used to start and stop the instance, sets
CATALINA_HOME to the proper runtime directory (which for tc Server could be Tomcat 6 or Tomcat 7), sets
CATALINA_BASE to the directory created by tcruntime-instance.sh, and then starts up (or shuts down) the application server.
There are a number of significant advantages to having separate
CATALINA_BASE directories. For one, it makes upgrading to a new release clean and easy. All you need to do is unpack the Tomcat distribution in its own directory and then set
CATALINA_HOME to this new directory before running startup.sh. (Note that this works only for point releases; the differences between major releases of Tomcat, such as 6 and 7, are too significant for this to work in most cases.) Similarly, you can simultaneously run multiple instances that use different minor versions of Tomcat.
Best Practice: Non-Root Ownership of CATALINA_BASE
This division of core runtime files from instance-specific configuration and web application files allows Tomcat users to follow a convention recommended as a best practice for tc Server: creating and running a Tomcat instance as a non-root user.
For tc Server, the default behavior of tcruntime-instance.sh is to create the
CATALINA_BASE directory at the root of the tc Server installation. By default, if tc Server was installed with yum/rpm (and by convention otherwise) that directory is /opt/vmware/vfabric-tc-server-standard/. This directory is owned by root, so unless an administrator modifies the owner and/or permissions on the directory, non-root users cannot create subdirectories therein. Which means that non-root users cannot use tcruntime-instance.sh to create an instance without specifying an alternative instance directory with the -i option. The parallel situation for Tomcat users is where the Tomcat distribution has been installed in a directory owned by root (let’s assume this directory is /usr/local/apache-tomcat-7.0.30). In a well-controlled environment, It makes sense for root (or another privileged user) to have exclusive write permission to the Tomcat install hierarchy. You don’t want random users to be able to modify the runtime used by every Tomcat instance. A consequence of this is that non-privileged users cannot run Tomcat without setting
CATALINA_BASE to a suitable directory for which they have write permission. This is because a running Tomcat needs to have permission to create directories and files (e.g. exploded war files in the webapps directory, log files in the logs directory) under
CATALINA_BASE is not set, it defaults to
CATALINA_HOME and since this directory has restricted access, non-privileged users cannot create the files and directories required for Tomcat to run.
Obviously, it is not good to require (or even allow) root to run Tomcat. As mentioned previously, the solution for tc Server users is to specify an alternative location for
CATALINA_BASE with the -i option to tcruntime-instance.sh. The rpm install creates a directory specifically for this purpose, /var/opt/vmware/vfabric-tc-server-standard, owned by the user tcserver, so a proper use of the command would be
prompt% ./tcruntime-instance.sh create node1 -i /var/opt/vmware/vfabric-tc-server-standard
This creates the directory /var/opt/vmware/vfabric-tc-server-standard/node1, which will be
CATALINA_BASE for this instance.
For Tomcat, one would designate a directory, such as /var/local/tomcat, as the standard location for all
CATALINA_BASE directories. This directory’s ownership, group, and permissions could be configured to allow members of a designated group, say tomcat, to create subdirectories therein. By convention, any
CATALINA_BASE directory would be located there, and Tomcat would be invoked with this sequence of commands:
prompt% export CATALINA_HOME=/usr/local/apache-tomcat-7.0.30
# node1 configured with proper subdirectories: bin, conf, logs, temp, webapps, work
prompt% export CATALINA_BASE=/var/local/tomcat/node1
Now that you’ve got a good overview, you can explore using multiple instances to more cleanly separate the core Tomcat binaries from the applications that will run in the Tomcat container.