As Lead Evangelist for Wavefront, a real-time analytics platform, I have the coolest job! I get to test the limits of the product and put together cool demos for its most advanced features. While playing around with our Distributed Tracing feature, I ran into an issue: I needed a way to reliably replay a set of known Zipkin-traces. And I needed to do that with the correct timing. In other words, I needed the stream of spans to appear exactly as if they’d come from a live application.
This challenge prompted me to open up my favorite Go editor and put together a little tool. It proved useful for both demo generation and the debugging of tracing tools, so we decided to share it with the open source community!
Zipkin Recording
The tool is made up of two parts. First, a trace recorder (zrecorder) acts as a very simple reverse proxy between my application and a backend system (in my case, Wavefront). The zrecorder program simply dumps all the Zipkin traffic it sees to a file and sends the traffic along unchanged to the backend.
When running zrecord, you specify the port to listen to, the file to dump the traces to, as well as the host and port to forward the traffic to. For example:
zrecord -file mytraces.txt -port 9411 -target http://myzipkintool:9411
Zipkin Replaying
The other part of the tool, zreplay, is (you guessed it) a player for recorded traces. It provides two modes of operation: A normal (or batch) mode and a real time mode.
In the normal mode, it simply sends traces to the backend as quickly as it can receive them. Since every span is timestamped, they should appear in the right order, provided the backend can handle backdating of spans correctly.
In real time mode, the tool preserves the relative timing between spans while adjusting the timestamps to make the stream of spans start at the time the tool was started. This allows you to truly simulate a running application generating traces. There is also a “-forever” flag that allows you to make the tool repeat the provided set of traces until the program is forcibly stopped.
When replaying traces, all trace and span IDs are re-generated to be unique for every run. If the “-forever” flag is specified, the IDs are regenerated every time the stream is restarted from the beginning of the recorded traces.
Since traces from a running program tend to take up a lot of storage space, there is also a “-gz” option that allows you to replay traces that have been gzipped.
To replay a set of recorded gzipped traces, you would run the following command:
zreplay -file traces.gz -target http://10.198.15.178:9411 -realtime -gz
Here is an example of replayed traces displayed in Wavefront. Of course, it also works with your favorite Open Source and commercial tracing tools, such as Jaeger, LightStep, inspectIT and Apache SkyWalking.
The zipkin-replay tool can be obtained from this Github repository. Feel free to try out the tool and, as always, we strongly encourage you to report bugs and suggest improvements. We also welcome contributions to the tool from the community!
Pontus Rydin works as Technical Evangelist Director for Wavefront by VMware. He has an extensive background in system monitoring and software development and is an active committer to open source projects such as Telegraf. Pontus resides in New Jersey and can be reached on Twitter (@prydin).
Stay tuned to the Open Source Blog and follow us on Twitter (@vmwopensource) for all the latest open source news and updates.
