# dotnet-counters Shows the Queue's Delta, Not Its Depth > A thread pool queue held at two thousand work items read 1986 through one provider and -13 through the other, in the same second, in the same process. - Published: 2026-09-13 - Tags: diagnostics, runtime, memory - Source: https://csharpmind.com/blog/dotnet-counters-shows-the-queues-delta-not-its-depth/ - Language: en-GB - Author: Callum Reeve --- ## TL;DR - A queue held at two thousand work items for fifteen seconds read 1986, 1990, 2000 and 1986 through the EventCounters provider and -11, 1, 11 and -13 through the Meter provider, over the same four seconds in the same process. - The Meter row is the first difference of the queue depth, because .NET 10 registers dotnet.thread_pool.queue.length with CreateObservableCounter rather than CreateObservableUpDownCounter. The tool subtracts successive totals, which is what a monotonic counter is for. - dotnet-counters prefers the Meter when a Meter and an EventSource share a name, so this is the row it shows unless the EventCounters prefix is typed. - A one-second interval is the floor, and it is a spot sample. A queue peaking at 497 twenty times per run read 0 in one hundred and fifteen consecutive readings across five runs, because the poll and the burst shared a period. - The registration is fixed on the runtime's release/11.0 branch under milestone 11.0.0 and was not backported, so the same dashboard row changes meaning on upgrade rather than on redeployment. --- A thread pool queue held at two thousand work items for fifteen seconds read 1, then -11, then 1, then 11, then -13. The same queue, over the same five seconds in the same process, read 1986, 1986, 1990, 2000 and 1986 through a second collector attached alongside the first. Both numbers came out of `dotnet-counters`, and the difference is which provider it was pointed at. The official guidance on diagnosing thread pool starvation says to run `dotnet-counters monitor -n ` and watch for "large values for `dotnet.thread_pool.queue.length`". On .NET 10 that row does not report how deep the queue is, and [where thread pool starvation actually comes from](/blog/where-thread-pool-starvation-actually-comes-from/) turns on reading exactly this counter against the thread count. ## The same queue, read through two providers The workload holds the queue at a fixed depth rather than letting it drain: a producer thread tops the pool back up to two thousand pending items every 20 ms for fifteen seconds, each item occupying a worker for 20 ms. A dedicated foreground thread samples `ThreadPool.PendingWorkItemCount` every 10 ms and is the ground truth below. Two `dotnet-counters` processes were attached to the same process id at once, one per provider, because a single session refuses the pair outright: *Using the same provider name with and without the EventCounters\ prefix in the counter list is not supported.* ```text true queue ThreadPool dotnet.thread_pool wall clock every 10 ms Queue Length .queue.length 06:52:04 1984-2000 2000 1996 06:52:05 1984-2000 1986 1 06:52:06 1981-2000 1986 -11 06:52:07 1980-2000 1990 1 06:52:08 1981-2000 2000 11 06:52:09 1980-2000 1986 -13 06:52:10 1979-2000 2000 15 06:52:11 1980-2000 1981 -2 06:52:12 1979-2000 1986 -16 06:52:13 1977-2000 1999 14 06:52:14 1976-2000 1995 -6 06:52:15 1980-2000 1993 2 ``` The two columns agree once, on the first reading — 2000 against 1996, the whole rise from an empty queue — and disagree on every reading after it. The right-hand column is the first difference of the left: a backlog that is neither growing nor shrinking produces small signed numbers centred on zero, which is also what an empty queue produces. When the producer stopped and the backlog drained, the three final readings were -268, -761 and -673 against a true queue falling from 1788 to nothing. ## The instrument the runtime registered `RuntimeMetrics.cs` in the `release/10.0` branch creates the thread pool instruments like this: ```csharp s_meter.CreateObservableCounter( "dotnet.thread_pool.queue.length", () => ThreadPool.PendingWorkItemCount, unit: "{work_item}", description: "The number of work items that are currently queued to be processed by the thread pool."); s_meter.CreateObservableUpDownCounter( "dotnet.timer.count", () => Timer.ActiveCount, unit: "{timer}", description: "The number of timer instances that are currently active. ..."); ``` Six lines apart, two callbacks that both return a level are registered with different factory methods. `Timer.ActiveCount` gets `CreateObservableUpDownCounter`; `ThreadPool.PendingWorkItemCount` gets `CreateObservableCounter`, and so does `ThreadPool.ThreadCount`. The runtime metrics reference documents both thread pool instruments as `UpDownCounter`, so the document and the shipping code disagree about what they are. Nothing downstream of that is wrong. An observable counter is monotonic by contract, its callback reports a cumulative total, and the only useful thing a collector can do with successive totals is subtract them — which is why the CSV column is headed `({work_item} / 1 sec)` and typed `Rate`. `dotnet-counters` is reporting the instrument faithfully; the instrument is the wrong kind. The runtime team agreed in February 2025, and the fix merged on 24 October 2025 under milestone 11.0.0. It was not backported: `release/11.0` creates both instruments with `CreateObservableUpDownCounter`, and `release/10.0` still creates them with `CreateObservableCounter`. ## What a one-second interval cannot resolve Choosing the provider that reports a level fixes the sign and not the sampling. `ThreadPool Queue Length` is one instantaneous reading per interval, and the interval is a long time. A second workload queues 500 items at the start of every second, each spinning 5 ms and allocating 64 KiB, which fills the queue to roughly 500 and drains it in about 200 ms. Sampled every 10 ms, one cycle looks like this: ```text t (ms) queue 0 474 88 250 179 32 267 0 354 0 442 0 529 0 616 0 704 0 792 0 880 0 ``` Across the run the queue reached 497 at its highest and was non-empty for 18 per cent of the samples. `ThreadPool Queue Length` reported 0 in all twenty-three readings, in five consecutive fresh processes — one hundred and fifteen readings, none of them non-zero. That is not bad luck. The burst period and the poll period are both one second, so a sample that lands in the idle gap once lands there every time. Changing only the burst period to 700 ms, leaving the amplitude and the interval alone, turned the same column into 81, 363, 199, 452, 9, 242 and 487 among the zeros. The counter did not improve; the workload stopped hiding behind it. Interval totals survive what spot samples do not. Over one of those runs `ThreadPool Completed Work Item Count` summed to exactly 10,000, the number of items the harness queued, and `Allocation Rate` summed to 656,471,696 bytes against the 656,873,304 the process reported at exit, short by the part of the last second the collector never sampled. What the interval removes is shape rather than quantity: the allocation rate read 32.8 MB/s every second, against roughly 152 MB/s sustained inside the 215 ms window that produced it, a factor of 4.6. ## What this changes Read the Counter Type column before the value. `Metric` is a level, `Rate` is a per-interval figure, and on .NET 10 a `Rate` row whose name describes a level — a queue length, a thread count — is a difference rather than a measurement. Every genuinely cumulative row is exact, in either provider: `dotnet.thread_pool.work_item.count` reported 500 per second and summed to the 10,000 items its process completed, and `dotnet.gc.heap.total_allocated` summed to 656,559,712 bytes against the 657,620,256 that process reported at exit. For a queue depth on .NET 10, type the prefix: `--counters 'EventCounters\System.Runtime[threadpool-queue-length]'`. It cannot be combined with the unprefixed provider, so a session that needs both instrument sets needs two sessions. `--refresh-interval` is parsed as `System.Int32`, so one second is the shortest interval that can be requested and 0.1 is rejected outright. Passing 0 does two different things: on the Meter path `AggregationManager.MinCollectionTimeSecs` is 0.1 and the request is rounded up to it, giving ten readings per second; on the EventCounters path `CounterGroup` reads an interval of zero or less as a signal to turn counters off, and the collector writes a CSV containing nothing but its header row. And the meaning of the row changes on upgrade rather than on redeployment. On the .NET 11 release branch the same counter name is published as an up-down counter, so a panel that has been quietly plotting deltas will start plotting depths, and a threshold tuned against the old behaviour will fire on the first busy second. Both numbers are real. Only one of them is the queue. --- **Measured on** .NET 10.0.12, SDK 10.0.401, macOS 26.6.2 arm64, 12 cores, Release configuration, `dotnet-counters` 10.0.745401. Ground truth is `ThreadPool.PendingWorkItemCount` read from a dedicated foreground thread every 10 ms; the tables are printed by a script that joins that sampler's CSV with the collector's CSV on the wall-clock second and are pasted as printed. The held-queue table is one process with two collector sessions attached at once; the burst figures are single-session runs, because a second session lengthened the drain from a median of 215 ms to 228 ms with a 464 ms outlier and the queue row stopped reading zero. A spot sample cannot report a maximum, so the true-queue column is the range the 10 ms sampler saw within that second and not a mean. The registration quoted is from `RuntimeMetrics.cs` on `release/10.0`, read on 13 September 2026 alongside `release/11.0`; the interval constants are from `AggregationManager` and `CounterGroup` in the same branch. --- ## Frequently asked ### Which provider does dotnet-counters use by default? The Meter. The tool's own help says that if the monitored application has both a Meter and an EventSource with the same name, the Meter is automatically preferred. Prefixing the provider with EventCounters\ selects the older path, and the two cannot appear in one --counters argument. ### Is the delta rendering a bug in dotnet-counters? No. The tool is displaying a monotonic counter as a rate, which is correct for that instrument kind. The defect is in the registration — .NET 10 creates dotnet.thread_pool.queue.length with CreateObservableCounter even though the documentation describes it as an UpDownCounter. ### Does a shorter refresh interval fix the missed spike? It narrows the gap and cannot close it. The option is parsed as an integer number of seconds, so one second is the shortest value that can be typed. Passing 0 gives ten readings per second on the Meter path and nothing at all on the EventCounters path. ### Which thread pool rows can be trusted on .NET 10? The genuinely monotonic ones. dotnet.thread_pool.work_item.count counts completed work items, so its per-interval rate is an exact interval total. The queue length and thread count rows are levels, and on .NET 10 only the EventCounters provider reports them as levels.