Virtual SAN ObserverIn the first installment of the Virtual SAN Performance Testing series, we reviewed benchmarking performance using synthetic I/O generation tool Iometer, automated by the VMware I/O Analyzer appliance. Using Iometer, or other synthetic I/O generation tools is frequently the first option for benchmarking selected, as it is an operationally light weight method to benchmark storage performance. But what if you want to be able to simulate real world workloads within your Virtual SAN cluster, without the burden of building out applications. That is where I/O trace files can come into play.

The most accurate way of evaluating a storage system is by running your target  applications in it and checking the performance. However, running actual applications are often not straight-forward especially when the application is complex and needs effort to tune it and set it up. While running actual applications lies on one end of the spectrum of evaluation methodologies, synthetic workloads lie on the other end. Synthetic workloads such as IOmeter are programs that issue requests in a pattern close to that of the applications. Running a synthetic workload for evaluation is very straight-forward, but often is proved to not accurately represent the characteristics of the real applications. I/O trace replay lies mid-way between these two techniques. Replaying an I/O trace is fairly easier and at the same time, accurately reproduces the application behavior to a large extent.


I/O Trace Files

So if trace file replay is a happy medium, you may be asking what exactly is a trace file, how do you replay trace files on Virtual SAN, and where do you get them? An I/O trace file is a collection of I/O requests issued by an application, that is collected at the layer below the file system. Within a vSphere environment, the vscsiStats tool (built into vSphere hosts) can be used to create custom application trace files.  A trace file can be of differing size and duration, depending on the capture time, and is in a .csv format. The easiest way to replay an I/O trace file in any vSphere  environment is utilizing VMware I/O Analyzer.

VMware  I/O Analyzer allows for scale out testing of Virtual SAN clusters using I/O  Traces. This can be done by uploading I/O Traces to the I/O Anlyzer controller, and using a built in script to run the I/O Trace on a pre-staged number of I/O Analyzer workers.

The I/O Analyzer virtual appliance comes with sample traces from a few common applications, such as Microsoft Exchange Mail Server, OLTP workload and others. You can also collect traces from applications of your choice in vSphere environments using the vscsistats tool, and upload them to I/O Analyzer Appliance. Collection with vscsistats will be covered in-depth in the next blog post of this series.

SNIA Trace Files Repository

If you want to explore utilizing  pre-created trace files that are not included with I/O Analyzer, another source of trace files is the SNIA Trace Files Repository. SNIA allows for access to a number of pre-created IO Traces, that are distributed using an open source license. The repository can be found at the following link.

I/O Analyzer Trace File Usage

To run a Trace Replay, the trace csv file must be uploaded to the IOAnalyzer Controller, and then using the traceReplayConfigGen script included with the I/O Analyzer appliance, a specified, equal number of workers on each host will execute the test for the user-specified duration. Before uploading your trace file to the I/O Analyzer controller, the I/O Analyzer workers should be cloned out onto your hosts, and have secondary drives larger than your trace minimum disk size requirements.  See the I/O Analyzer User guide for more information on adding workers to hosts.

Trace File Replay Steps

Upload a Trace Replay File

  1. Determine the IP Address of the IOAnalyzer Controller
  2. Navigate to http://IO.Analyzer.IP.Address:8180/IOAnalyzer/#menu
  3. Click on Upload VSCSI Trace
  4. Provide the Trace Name and Optional Trace Info
  5. Select the File from your Remote or Local Machine
  6. Click Upload Local File

Verify Trace Replay File Uploaded

  1. Click on the Home icon
  2. Click on VSCSI Trace Characteristics
  3. Select the Trace Name from the drop-down

Generate a Trace Replay Run Configuration

  1. Use ssh client to log into the IOAnalyzer Controller
  2. Cd to /root/ioAnalyzerScripts
  3. Execute the script per below:

–testname is the RunConfig name that will be used when executing a Test Replay with the workers.

–password is the password to all of the hosts in the cluster

–vmcount is the number of per host number vm’s to add to the RunConfig.  For example –vmcount 5 workers on an 8 node cluster means 40 workers totla

–trace is the trace file which all of the workers will run.

–vmprefix is the prefix of the IOAnalyzer worker vm’s, example w101

–hosts lists all of the hosts that will host workers running the Trace Replay.  Do not use a space in between the commas.

 Example command below

localhost:~/ioAnalyzerScripts # ./ –testname  exampleconfig_40w  –password vmware  –vmcount 5 –trace exampletrace –vmprefix  w101 –hosts,,,,,,,

 Verify that the Workload Config was created correctly

  1. Navigate to the IOAnalyzer URL
  2. Click on the Gear icon
  3. Under Workload Configuration
    1. Select the new Workload Configuration, example exampleconfig_40w in the dropdown
    2. Click the Load Config Button
    3. Verify that the correct number of workers were loaded with the correct Trace Replay


I/O Analyzer replay of traces files allows for an efficient way to perform scale out testing using real word application workload replays. In my next post, I will delve into creating custom trace replay files for scale out testing using I/O Analyzer.

  1. Virtual SAN Performance Testing Part I – Utilizing I/O Analyzer with Iometer
  2. Virtual SAN Performance Testing Part II – Utilizing I/O Analyzer with Application Trace Files
  3. Virtual SAN Performance Testing Part III – Utilizing Custom Application Trace files
  4. Virtual SAN Performance Testing Part IV – Analyzing Performance Results