Collect metrics - .NET (2023)

This article applies to: ✔️ .NET Core 3.1 and later versions ✔️ .NET Framework 4.6.1 and later versions

Instrumented code can record numeric measurements, but the measurements usually need to be aggregated, transmitted,and stored to create useful metrics for monitoring. This process of aggregating, transmitting, and storing the data iscalled collection. In this tutorial, we will show several examples on how to collect metrics:

• Populating metrics in Grafana with OpenTelemetry and Prometheus.
• Viewing metrics in real time with the dotnet-counters command-line tool.
• Creating a custom collection tool using the underlying .NET MeterListener API.

For more information about custom metric instrumentation and an overview of instrumentation options, see Compare metric APIs.

Create an example application

Prerequisites: .NET Core 3.1 SDK or a later version

Before metrics can be collected, we need to produce some measurements. For simplicity, we will create a small app that hassome trivial metric instrumentation. The .NET runtime also has various metrics built-in. For more information about creating new metrics using theSystem.Diagnostics.Metrics.Meter API shown here, seethe instrumentation tutorial.

dotnet new consoledotnet add package System.Diagnostics.DiagnosticSource

Replace the code of Program.cs with:

using System;using System.Diagnostics.Metrics;using System.Threading;class Program{ static Meter s_meter = new Meter("HatCo.HatStore", "1.0.0"); static Counter<int> s_hatsSold = s_meter.CreateCounter<int>("hats-sold"); static void Main(string[] args) { Console.WriteLine("Press any key to exit"); while(!Console.KeyAvailable) { // Pretend our store has a transaction each second that sells 4 hats Thread.Sleep(1000); s_hatsSold.Add(4); } }}

View metrics with dotnet-counters

dotnet-counters is a simple command-line tool that can view live metrics for any .NET Core application ondemand. It doesn't require any advance setup, which can make it useful for ad-hoc investigations or to verify that metricinstrumentation is working correctly. It works with both System.Diagnostics.Metrics basedAPIs and EventCounters.

If the dotnet-counters tool is not already installed, use the SDK to install it:

> dotnet tool update -g dotnet-countersYou can invoke the tool using the following command: dotnet-countersTool 'dotnet-counters' (version '5.0.251802') was successfully installed.

While the example app is still running, list the running processes in a second shell to determine the process ID:

> dotnet-counters ps 10180 dotnet C:\Program Files\dotnet\dotnet.exe 19964 metric-instr E:\temp\metric-instr\bin\Debug\netcoreapp3.1\metric-instr.exe

Find the ID for the process name that matches the example app and have dotnet-counters monitor all metrics from the"HatCo.HatStore" meter. The meter name is case-sensitive.

> dotnet-counters monitor -p 19964 HatCo.HatStorePress p to pause, r to resume, q to quit. Status: Running[HatCo.HatStore] hats-sold (Count / 1 sec) 4

We can also run dotnet-counters specifying a different set of metrics to see some of the built-in instrumentationfrom the .NET runtime:

> dotnet-counters monitor -p 19964 System.RuntimePress p to pause, r to resume, q to quit. Status: Running[System.Runtime] % Time in GC since last GC (%) 0 Allocation Rate (B / 1 sec) 8,168 CPU Usage (%) 0 Exception Count (Count / 1 sec) 0 GC Heap Size (MB) 2 Gen 0 GC Count (Count / 1 sec) 0 Gen 0 Size (B) 2,216,256 Gen 1 GC Count (Count / 1 sec) 0 Gen 1 Size (B) 423,392 Gen 2 GC Count (Count / 1 sec) 0 Gen 2 Size (B) 203,248 LOH Size (B) 933,216 Monitor Lock Contention Count (Count / 1 sec) 0 Number of Active Timers 1 Number of Assemblies Loaded 39 ThreadPool Completed Work Item Count (Count / 1 sec) 0 ThreadPool Queue Length 0 ThreadPool Thread Count 3 Working Set (MB) 30

For more information about the tool, see the dotnet-counters.To learn more about metrics that are available out of the box in .NET, see built-in metrics.

View metrics in Grafana with OpenTelemetry and Prometheus

Prerequisites

• .NET Core 3.1 SDK or a later version

Overview

OpenTelemetry is a vendor-neutral open-source project supported by theCloud Native Computing Foundation that aims to standardize generating and collecting telemetry forcloud-native software. The built-in platform metric APIs are designed to be compatible with thisstandard to make integration straightforward for any .NET developers that wish to use it. At the time of writing, support forOpenTelemetry metrics is relatively new, but Azure Monitorand many major APM vendors have endorsed it and have integration plans underway.

This example shows one of the integrations available now for OpenTelemetry metrics using the popular OSSPrometheus and Grafana projects. The metrics data will flow like this:

1. The .NET metric APIs collect measurements from our example application.
2. The OpenTelemetry library running inside the same process aggregates these measurements.
3. The Prometheus exporter library makes the aggregated data available via an HTTPmetrics endpoint. 'Exporter' is what OpenTelemetry calls the libraries that transmittelemetry to vendor-specific backends.
4. A Prometheus server, potentially running on a different machine, polls themetrics endpoint, reads the data, and stores it in a database for long-term persistence.Prometheus refers to this as 'scraping' an endpoint.
5. The Grafana server, potentially running on a different machine, queries the datastored in Prometheus and displays it to engineers on a web-based monitoring dashboard.

Configure the example application to use OpenTelemetry's Prometheus exporter

Add a reference to the OpenTelemetry Prometheus exporter to the example application:

dotnet add package OpenTelemetry.Exporter.Prometheus --version 1.2.0-beta1

Modify the code of Program.cs so that it contains the extra code to configure OpenTelemetry at the beginning of Main():

using System;using System.Diagnostics.Metrics;using System.Threading;using OpenTelemetry;using OpenTelemetry.Metrics;class Program{ static Meter s_meter = new Meter("HatCo.HatStore", "1.0.0"); static Counter<int> s_hatsSold = s_meter.CreateCounter<int>(name: "hats-sold", unit: "Hats", description: "The number of hats sold in our store"); static void Main(string[] args) { using MeterProvider meterProvider = Sdk.CreateMeterProviderBuilder() .AddMeter("HatCo.HatStore") .AddPrometheusExporter(opt => { opt.StartHttpListener = true; opt.HttpListenerPrefixes = new string[] { $"http://localhost:9184/" }; }) .Build(); Console.WriteLine("Press any key to exit"); while(!Console.KeyAvailable) { // Pretend our store has a transaction each second that sells 4 hats Thread.Sleep(1000); s_hatsSold.Add(4); } }}

AddMeter("HatCo.HatStore") configures OpenTelemetry to transmit all the metrics collected by the Meter our app defined.AddPrometheusExporter(...) configures OpenTelemetry to expose Prometheus' metrics endpoint on port 9184 and to usethe HttpListener. See the OpenTelemetry documentationfor more information about OpenTelemetry configuration options, in particular, alternative hosting options that are useful for ASP.NET applications.

Run the example app and leave it running in the background.

> dotnet runPress any key to exit

Set up and configure Prometheus

Follow the Prometheus first steps to set up your Prometheus serverand confirm it is working.

Modify the prometheus.yml configuration file so that Prometheus will scrape the metrics endpoint that our example app isexposing. Add this text in the scrape_configs section:

 - job_name: 'OpenTelemetryTest' scrape_interval: 1s # poll very quickly for a more responsive demo static_configs: - targets: ['localhost:9184']

If you are starting from the default configuration, then scrape_configs should now look like this:

scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: "prometheus" # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ["localhost:9090"] - job_name: 'OpenTelemetryTest' scrape_interval: 1s # poll very quickly for a more responsive demo static_configs: - targets: ['localhost:9184']

Reload the configuration or restart the Prometheus server, then confirm that OpenTelemetryTest is in the UPstate in the Status > Targets page of the Prometheus web portal.

On the Graph page of the Prometheus web portal, enter hats_sold in the expression text box. In the graph tab, Prometheus shouldshow the steadily increasing value of the "hats-sold" Counter that is being emitted by our example application.

If the Prometheus server hasn't been scraping the example app for long, you may need to wait a short while for data to accumulate.You can also adjust the time range control in the upper left to "1m" (1 minute) to get a better view of very recent data.

Show metrics on a Grafana dashboard

1. Follow the standard instructions to install Grafana andconnect it to a Prometheus data source.

2. Create a Grafana dashboard by clicking the + icon on the left toolbar in the Grafana web portal, then select Dashboard. In the dashboardeditor that appears, enter 'Hats Sold/Sec' as the Title and 'rate(hats_sold[5m])' in the PromQL expression field. It should look like this:

3. Click Apply to save and view the simple new dashboard.

   (Video) How to collect the code coverage of your tests in .NET

The .NET MeterListener API allows creating custom in-process logic to observe the measurementsbeing recorded by System.Diagnostics.Metrics.Meter. For guidance creating customlogic compatible with the older EventCounters instrumentation, see EventCounters.

Modify the code of Program.cs to use MeterListener like this:

using System;using System.Collections.Generic;using System.Diagnostics.Metrics;using System.Threading;class Program{ static Meter s_meter = new Meter("HatCo.HatStore", "1.0.0"); static Counter<int> s_hatsSold = s_meter.CreateCounter<int>(name: "hats-sold", unit: "Hats", description: "The number of hats sold in our store"); static void Main(string[] args) { using MeterListener meterListener = new MeterListener(); meterListener.InstrumentPublished = (instrument, listener) => { if(instrument.Meter.Name == "HatCo.HatStore") { listener.EnableMeasurementEvents(instrument); } }; meterListener.SetMeasurementEventCallback<int>(OnMeasurementRecorded); meterListener.Start(); Console.WriteLine("Press any key to exit"); while(!Console.KeyAvailable) { // Pretend our store has a transaction each second that sells 4 hats Thread.Sleep(1000); s_hatsSold.Add(4); } } static void OnMeasurementRecorded<T>(Instrument instrument, T measurement, ReadOnlySpan<KeyValuePair<string,object>> tags, object state) { Console.WriteLine($"{instrument.Name} recorded measurement {measurement}"); }}

When run, the application now runs our custom callback on each measurement:

> dotnet runPress any key to exithats-sold recorded measurement 4hats-sold recorded measurement 4hats-sold recorded measurement 4hats-sold recorded measurement 4...

Let's break down what happens in the example above.

using MeterListener meterListener = new MeterListener();

First we created an instance of the MeterListener, which we will use to receive measurements.

meterListener.InstrumentPublished = (instrument, listener) =>{ if(instrument.Meter.Name == "HatCo.HatStore") { listener.EnableMeasurementEvents(instrument); }};

Here we configured which instruments the listener will receive measurements from.InstrumentPublished is a delegate that will be invoked anytime a newinstrument is created within the app. Our delegate can examine the instrument, such as checking the name, the Meter, or any otherpublic property to decide whether to subscribe. If we do want to receive measurements from this instrument, then we invokeEnableMeasurementEvents to indicate that. If your code has another wayto obtain a reference to an instrument, it's legal to invoke EnableMeasurementEvents() at any time with that reference, but this isprobably uncommon.

meterListener.SetMeasurementEventCallback<int>(OnMeasurementRecorded);...static void OnMeasurementRecorded<T>(Instrument instrument, T measurement, ReadOnlySpan<KeyValuePair<string,object>> tags, object state){ Console.WriteLine($"{instrument.Name} recorded measurement {measurement}");}

Next we configured the delegate that is invoked when measurements are received from an instrument by callingSetMeasurementEventCallback. The generic parameter controls which data typeof measurement will be received by the callback. For example, a Counter<int> generates int measurements whereas aCounter<double> generates double measurements. Instruments can be created with byte, short, int, long,float, double, and decimal types. We recommend registering a callback for every data type unless you have scenario-specific knowledge that not all data types will be needed, such as in this example. Making repeated calls toSetMeasurementEventCallback() with different generic arguments may appear a little unusual. The API is designed this wayto allow MeterListeners to receive measurements with extremely low performance overhead, typically just a few nanoseconds.

When MeterListener.EnableMeasurementEvents() was called initially, there was an opportunity to provide a state object asone of the parameters. That object can be anything you want. If you provide a state object in that call, then it will bestored with that instrument and returned to you as the state parameter in the callback. This is intended both as aconvenience and as a performance optimization. Often listeners need to create an object for each instrument that willstore measurements in memory and have code to do calculations on those measurements. Although you could create a Dictionarythat maps from the instrument to the storage object and look it up on every measurement, that would be much slower thanaccessing it from state.

meterListener.Start();

Once the MeterListener is configured, we need to start it to trigger callbacks to begin. The InstrumentPublisheddelegate will be invoked for every pre-existing Instrument in the process. In the future, any newly created Instrumentwill also trigger InstrumentPublished to be invoked.

using MeterListener meterListener = new MeterListener();

Once we are done listening, disposing the listener stops the flow of callbacks and releases any internal referencesto the listener object. The using keyword we used when declaring meterListener causes Dispose() to be called automaticallywhen the variable goes out of scope. Be aware that Dispose() is only promising that it won't initiate new callbacks. Because callbacksoccur on different threads, there may still be callbacks in progress after the call to Dispose() returns. If you need aguarantee that a certain region of code in your callback isn't currently executing and will never execute again in the future,then you will need some additional thread synchronization to enforce that. Dispose() doesn't include the synchronizationby default because it adds performance overhead in every measurement callback—and MeterListener is designed as a highlyperformance conscious API.