Weathervane is a performance benchmarking tool developed at VMware. It lets you assess the performance of your virtualized or cloud environment by driving a load against a realistic application and capturing relevant performance metrics. You might use it to compare the performance characteristics of two different environments, or to understand the performance impact of some change in an existing environment.
Weathervane is very flexible, allowing you to configure almost every aspect of a test, and yet is easy to use thanks to tools that help prepare your test environment and a powerful run harness that automates almost every aspect of your performance tests. You can typically go from a fresh start to running performance tests with a large multi-tier application in a single day.
Weathervane supports a number of advanced capabilities, such as deploying multiple independent application instances, deploying application services in containers, driving variable loads, and allowing run-time configuration changes for measuring elasticity-related performance metrics.
Weathervane has been used extensively within VMware, and is now open source and available on GitHub at https://github.com/vmware/weathervane.
The rest of this blog gives an overview of the primary features of Weathervane.
Weathervane Overview
Weathervane is an application-level benchmarking tool. It allows you to place a controlled load on a computing environment by deploying a realistic application, and then simulating users interacting with the application. In the case of Weathervane, the application is a scalable Java web application which implements a real-time auction web site, and the environment can be anything from a single bare-metal server to a large virtualized cluster of servers or a public or private cloud. You can use collected metrics to evaluate the performance of the environment or to investigate the effect of changes in the environment. For example, you could use Weathervane to compare the performance of different cloud environments, or to evaluate the impact of changing storage technologies on application-level performance.
Weathervane consists of three main components and a number of supporting tools:
- The Auction application to be deployed on the environment under test.
- A workload driver that can drive a realistic and repeatable load against the application.
- A run harness which automates the process of executing runs and collecting results, logs, and relevant performance data.
- Supporting tools include scripts to set up an operating system instance with all of the software needed to run Weathervane, to create Docker images for the application services, and to load and prepare the application data needed for a run of the benchmark.
Figure 1 shows the logical layout of the main components of a Weathervane deployment. Additional background about the components of Weathervane can be found at https://blogs.vmware.com/performance/2015/03/introducing-weathervane-benchmark.html.
Figure 1 Weathervane Deployment
The design goal for Weathervane has been to provide flexibility so that you can adapt it to suit the needs a wide range of performance evaluation tasks. You can customize almost every aspect of a Weathervane deployment. You can vary …
- … the number of instances in each service tier. For example, there can be any number of load balancer, application server, or web server nodes. This allows you to create very small or very large configurations as needed.
- … the number of tiers. For example, it is possible to omit the load-balancer and/or web server tiers if only a small application deployment is needed.
- … the implementation to be used for each service. For example, Weathervane currently supports PostgreSQL and MySQL as the transactional database, and Apache Httpd and Nginx as the web server. Because Weathervane is open source, you can add additional implementations of a service type if desired.
- … the tuning and configuration of the services. The most common performance and configuration tunings for each service tier can be set in the Weathervane configuration file. The run harness will then apply the tunings to each service instance automatically when you perform a run of the benchmark.
The Weathervane run harness makes it easy for you to take advantage of this flexibility by managing the complexity involved in configuring and starting the application, running the workload, and collecting performance results. In many cases, you can make complex changes in a Weathervane deployment with just a few simple changes in a configuration file.
Weathervane also provides advanced features that allow you to evaluate the performance impact of many important issues in large virtualized and cloud environments.
- You can run the service instances of the Auction application directly on the OS, either in a virtual machine or on a bare-metal server, or deploy them in Docker containers. Weathervane comes with scripts to create Docker images for all of the application services.
- You can run and drive load against multiple independent instances of the Auction application. This is useful if you want to investigate the interactions between multiple independent applications, or when it is necessary to drive loads larger than can be handled by a single application instance. The configuration and load for each instance can be specified independently.
- You can specify a user load for the application instances that varies over the course of run. This allows you to investigate performance issues related to bursty loads, cyclical usage patterns, and, as discussed next, the impact of application elasticity.
- The Action application supports changing the number of instances of some service tiers at run-time to support application-level elasticity (https://en.wikipedia.org/wiki/Elasticity_(cloud_computing)), and the study of elasticity-related performance metrics. Weathervane currently includes a scheduled elasticity-service, which allows you to specify changes in the application configuration over time. When used in combination with the time-varying loads, this enables the investigation of some elasticity-related performance issues. Future implementations of the elasticity service will use real-time monitoring to make decisions about configuration changes.
Over the coming months, we will be publishing additional posts demonstrating the use of these features.
Future Direction
We intend to continue to grow and improve Weathervane’s applicability and ease of use. We also plan to focus on improving its usefulness as a platform for examining the meaning of performance evaluation in the cloud. As an open source project, we invite the wider performance community to not only use it, but to participate in extending it and shaping its future direction. Enhancements to Weathervane may include adding new performance metrics, better metric reporting and real-time monitoring, support for additional services such as cloud-vendor specific databases and load balancers, and even adding additional applications to be deployed on the environment under test. Visit the Weathervane GitHub repository at https://github.com/vmware/weathervane for more information about getting involved.
The Weathervane team would like to thank the VMware Open-Source Program Office, https://blogs.vmware.com/opensource/, and VMware’s commitment to open-source software, for helping to make this release possible.