Distributed tracing is a critical piece of application observability. But, the sheer number of traces containing a ton of information can be overwhelming. In this blog, I’ll show you concrete examples of how the Wavefront Query Language can be applied to distributed tracing and quickly get you answers to all your questions, significantly reducing troubleshooting time.

Sample Microservices Application: BeachShirts

As you can see in the above diagram, when someone tries to order shirts, the end to end workflow is depicted by the numbered sequence of operations. Here’s how an end to end trace looks like for the “orderShirts” workflow:

Wavefront Query Language for Distributed Tracing

In the above diagram, notice that the end to end trace is taking 11.74 seconds. When it comes to microservices, this latency is unacceptable. At first glance, it’s easy to blame the shopping service (“ShoppingWebResource.orderShirts” operation) since it takes 11.56 seconds. But when you look closer at the critical path highlighted with the orange line, then it’s obvious that packaging service (“Packaging.giftWrap” operation) is taking the bulk of that time and slowing down shopping service. This key piece of information is something you would be missing without Distributed Tracing.

So now that we agree that tracing is invaluable, how can you search for those traces? Wavefront has built a very intuitive Traces Browser that uses Query Builder under the hood to show traces based on your search criteria. But what if you are a more advanced user who wants to leverage the Wavefront query language for tracing? You can easily switch from the Traces Browser mode from Query Builder to Query Editor mode.

Let’s take a step by step approach to crafting a traces() query. Let’s start with the basic use case and then work up to the more advanced use case.

Use case 1: Get Me All the Traces for orderShirts Operation

In order to do this, you can either use the Query Builder or the Query Editor. This is how you would search for orderShirts operation via Query Builder.

This will result in the following query:

limit(100, traces("beachshirts.shopping.ShoppingWebResource.orderShirts"))

You could have also used the Query Editor and typed in the above query manually. This is what the output would look like:

Notice that most of the traces are in the order of seconds as opposed to milliseconds. These traces are really taking long, almost ~10 seconds. What if you want to search for only long traces? How would you go about it?

Use case 2: Get me all traces longer than a certain threshold

Let’s say you only want to search for traces greater than a certain min (> 10 seconds) but less than some max (< 20 seconds). You can do this with the Duration filter.

It will result in the following query:

limit(100, lowpass(20s, highpass(10s, traces("beachshirts.shopping.ShoppingWebResource.orderShirts"))))

But wait for a second. This isn’t that helpful because as per the above trace, you’ll notice that the bottleneck lies in ‘Packaging.giftWrap’ operation. Here Packaging.gitfWrap is the critical span highlighted with the orange line. How would you narrow down traces where a certain span is a bottleneck? What if you knew that and wanted to search for all long traces with that span? How would you go about it?

Use case 3: Get me all the long traces for Packaging.gitWrap operation

You would issue a similar query but this time select ‘Packaging.giftWrap’ operation. The query would be as follows:

limit(100, lowpass(20s, highpass(10s, traces(spans("beachshirts.packaging.Packaging.giftWrap")))))

When you do this you would still get long traces where Packaging.giftWrap might not be the bottleneck. How do we narrow this down even further?

Use case 4: Filter all traces where a certain span (Packaging.gitWrap) is greater than a certain threshold

What you really want to do is search for traces where a Packaging.giftWrap is taking longer than a certain threshold. If you want to narrow down traces where Packaging.giftWrap is taking longer than 7 seconds, then you could wrap that span in a highpass filter. This is how that query would look like:

limit(100, lowpass(20s, highpass(10s, traces(highpass(7s, spans("beachshirts.packaging.Packaging.giftWrap"))))))

Use case 5: Only show me traces where Packaging.gitWrap had an error

Wait, we’re not done yet. What if you want to search for traces where Packaging.giftWrap had an error. Easy! Just apply error=true filter for that span, i.e. spans(“beachshirts.packaging.Packaging.giftWrap”, “error”=”true”)

You can narrow it down even further if you want to apply some tags to Packaging.giftWrap operation. Say if you want to retrieve traces where that Packaging.giftWrap is applicable for Production environment and location=Palo Alto, then your final query would look like this:

limit(100, lowpass(20s, highpass(10s, traces(highpass(7s, spans("beachshirts.packaging.Packaging.giftWrap",
"error"="true" and "env"="Production" and "location"="Palo Alto"))))))

Conclusion

Tracing is most context-rich instrumented data that you can extract from your microservices application. At the same time, this data can be really verbose and noisy.

The Wavefront Query Language for traces provides the most flexible, powerful and extensible way to narrow down your search radius and get to the outlier traces faster. For more information, refer to our documentation and checkout our free trial today!

Get Started with Wavefront Follow @sushantdewan Follow @WavefrontHQ

The post How to Search for Outlier Traces: A Guide to Wavefront Query Language for Distributed Tracing appeared first on Wavefront by VMware.