Understanding Istio Telemetry v2

It’s been a while since I’ve blogged, and just like other posts in the past, this one is meant as a way to dig into something and for me to catalog my own thoughts for later. While digging into some issues for some of our Istio customers as well as for a chapter in my upcoming book, Istio in Action, I found myself knee-deep in the Istio telemetry v2 functionality. Let’s see how it works.

We will use source code from https://github.com/christian-posta/istio-telemetry-v2 for this blog.

Istio telemetry v2

Istio telemetry v2 is a combination of data-plane extensions (ie, Envoy extensions) and an programable API to allow operators to tune, customize, and even create “service-level” metrics within the proxy. This “v2” status replaces a previous implementation based on an out-of-band integration engine called Mixer.

There are three main concepts in the telemetry v2 functionality that you should understand to fully wrap your ahead around what it’s doing and how to customize it:

• Metrics
• Dimensions
• Attributes

A metric is a counter, gauge, or histogram/distribution of telemetry signals between service calls (inbound/outbound). For example, some of the Istio standard metrics are:

• istio_requests_total a COUNTER measuring total number of requests
• istio_request_duration_milliseconds a DISTRIBUTION measuring latency of requests
• istio_request_bytes a DISTRIBUTION which measure http request body sizes
• istio_response_bytes a DISTRIBUTION which measures http response body sizes

For the istio_requests_total metric, we count the total number of requests that have come through. The interesting bit is that a metric can have various dimensions which are additional properties that give more depth and insight for a particular metric.

From the docs, you can see for example, the istio_requests_total metric has some out of the box dimensions. Here’s an example of those dimensions:

This means we’ve seen 5 requests from the web-api app to the recommendation app that have a response_code of HTTP 200. If any of these dimensions are different, we’ll see a new entry for this metric. For example, if there are any HTTP 500 response codes, we’d see this in a different line (some dimensions left out for brevity):

The last important bit of detail is where these dimensions come from. To answer this, we need to understand attributes and CEL expressions. In it’s simplest form a dimension gets its values at runtime from attributes that come from Envoy’s underlying attributes or from Istio’s peer-metadata filter.

For example, let’s see the Request Attributes that come from Envoy:

For example, to map an attribute to a dimension, we can configure the metric (we’ll see that in the next section) like this:

As stated earlier, there is a wealth of available attributes out of the box from Envoy as well as from Istio’s Peer Metadata plugin. Please check the respective docs. We can even create our own derivative attributes to use for a particular dimension which we’ll see in the next section.

Metrics in Action

Let’s see how this all works with an example. You can follow along with the source at https://github.com/christian-posta/istio-telemetry-v2.

First, set up and deploy the sample applications. The sample applications show a call graph between three different services:

web-api –> recommendation –> purchase-history

We have configured the purchase-history service to return errors on 50% of the calls (HTTP 500) and for the rest, to return a latency of P50 of 750ms (variance of 100ms). We can easily make some sample calls with the following command (you should try run this a few times):

Now, let’s evaluate the metrics from the recommendation service to see what has been captured and for what dimensions:

We should see something like:

We can see four different entries for the istio_requests_total along a couple different dimensions (some dimensions removed for brevity). We see differences in the reporter, response_code, source_workload, and destiation_workload dimensions.

We can see a latency distribution for the requests with the istio_request_duration_milliseconds metric

Customizing metrics

We can also customize what dimensions are included in a particular metric. In fact, there is already an out of the box example for how to configure these metrics. When we install Istio, a few EnvoyFilters are installed which configure metrics.

The one we’re interested in is the stats-filter-* EnvoyFilter. If we take a look at the stats-filter-1.9 we see an EnvoyFilter definition but the salient part is here:

This EnvoyFilter is used to ADD a new configuration to the Envoy Http Connection Manager and the chain of filters used to process an HTTP request. Note there are multiple configuration sections in this EnvoyFilter because we configure both the INBOUND as well as OUTBOUND paths. Specifically this is added toward the end of the chain BEFORE the router filter (this is important… the router should be the last filter in this chain). The important config bits are the following:

This configuration stanza corresponds to the Istio docs here and if you’re really interested, corresponds to the proto in the stats extension here.

Specifically what gets configured here is the cluster dimensions for ALL of the standard Istio metrics (it’s ALL because we don’t explicitly name a metric here).

Let’s change the metrics a bit. We can edit the stats-filter directly, or we can create a different EnvoyFilter that augments the stats-filter with our new configuration. From the source code for this blog, see the customize-metric.yaml file for the full contents:

In this configuration we’ve added two new dimensions called posta and posta_two and we use attributes from the previous section to populate them.

If we reviewed our metric at this point, we’d see some discrepancies. The posta and posta_two dimension is not known to our proxy so before we can use it, we need to expose it. This is because these new dimensions are not in the default tag list that Istio knows about. Let’s add the following annotation to our recommendation Deployment pod spec:

This exposes the metric dimensions correctly.

Now let’s place a few calls:

Now if we review our istio_requests_total metric we should see the new dimensions:

Creating new metrics

The last thing we’ll look at in this blog is creating a new metric. To do that, we need to specify a metric definition in the configuration for the stats plugin. Something like this would work to create a new metric called posta_metric:

This is a very simple metric of type COUNTER which just counts requests when they come in (just like istio_requests_total). However, the value field is actually a string where you can place a CEL expression that evaluates some attributes; just note that this expression should evaluate to an integer.

Let’s apply the create-new-metric.yaml from our source code repo:

Just like we exposed extra dimensions on the recommendation deployment in the previous step, we will need to expose this new metric with the statsInclusionPrefixes annotation:

Note that even though we called the metric posta_metric it gets a prefix of istio_ anyway.

Now let’s send some more traffic:

Now if we review our istio_requests_total metric we should see the new dimensions:

Note there are no dimensions for this metric! Just like we customized the dimensions for metrics in the previous section, we could do something like this:

Note, when we name the metric explicitly, we DON’T need to use the prefix istio_ as it will understand it by default.

Creating your own attributes

Hopefully this blog as gone into enough detail about understanding metrics and Istio’s telemetry v2. Armed with this information, you should now be able to see the Istio docs about generating your own attributes so you can use those in dimensions.

For more information

I cover Istio telemetry v2 deeply in chapter 7 of Istio in Action. Also check the community Istio docs. If you’re deploying Istio and need help, please reach out to me (@christianposta) or ceposta on CNCF/Kubernetes/Istio/Solo.io slack.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.