dotnet-counters Shows the Queue's Delta, Not Its Depth

A thread pool queue held at two thousand work items for fifteen seconds read 1, then -11, then 1, then 11, then -13. The same queue, over the same five seconds in the same process, read 1986, 1986, 1990, 2000 and 1986 through a second collector attached alongside the first.

Both numbers came out of dotnet-counters, and the difference is which provider it was pointed at. The official guidance on diagnosing thread pool starvation says to run dotnet-counters monitor -n <app> and watch for “large values for dotnet.thread_pool.queue.length”. On .NET 10 that row does not report how deep the queue is, and where thread pool starvation actually comes from turns on reading exactly this counter against the thread count.

The same queue, read through two providers

The workload holds the queue at a fixed depth rather than letting it drain: a producer thread tops the pool back up to two thousand pending items every 20 ms for fifteen seconds, each item occupying a worker for 20 ms. A dedicated foreground thread samples ThreadPool.PendingWorkItemCount every 10 ms and is the ground truth below. Two dotnet-counters processes were attached to the same process id at once, one per provider, because a single session refuses the pair outright: Using the same provider name with and without the EventCounters\ prefix in the counter list is not supported.

                 true queue   ThreadPool     dotnet.thread_pool
  wall clock    every 10 ms   Queue Length     .queue.length
  06:52:04      1984-2000           2000            1996
  06:52:05      1984-2000           1986               1
  06:52:06      1981-2000           1986             -11
  06:52:07      1980-2000           1990               1
  06:52:08      1981-2000           2000              11
  06:52:09      1980-2000           1986             -13
  06:52:10      1979-2000           2000              15
  06:52:11      1980-2000           1981              -2
  06:52:12      1979-2000           1986             -16
  06:52:13      1977-2000           1999              14
  06:52:14      1976-2000           1995              -6
  06:52:15      1980-2000           1993               2

The two columns agree once, on the first reading — 2000 against 1996, the whole rise from an empty queue — and disagree on every reading after it. The right-hand column is the first difference of the left: a backlog that is neither growing nor shrinking produces small signed numbers centred on zero, which is also what an empty queue produces. When the producer stopped and the backlog drained, the three final readings were -268, -761 and -673 against a true queue falling from 1788 to nothing.

The instrument the runtime registered

RuntimeMetrics.cs in the release/10.0 branch creates the thread pool instruments like this:

s_meter.CreateObservableCounter(
    "dotnet.thread_pool.queue.length",
    () => ThreadPool.PendingWorkItemCount,
    unit: "{work_item}",
    description: "The number of work items that are currently queued to be processed by the thread pool.");

s_meter.CreateObservableUpDownCounter(
    "dotnet.timer.count",
    () => Timer.ActiveCount,
    unit: "{timer}",
    description: "The number of timer instances that are currently active. ...");

Six lines apart, two callbacks that both return a level are registered with different factory methods. Timer.ActiveCount gets CreateObservableUpDownCounter; ThreadPool.PendingWorkItemCount gets CreateObservableCounter, and so does ThreadPool.ThreadCount. The runtime metrics reference documents both thread pool instruments as UpDownCounter<T>, so the document and the shipping code disagree about what they are.

Nothing downstream of that is wrong. An observable counter is monotonic by contract, its callback reports a cumulative total, and the only useful thing a collector can do with successive totals is subtract them — which is why the CSV column is headed ({work_item} / 1 sec) and typed Rate. dotnet-counters is reporting the instrument faithfully; the instrument is the wrong kind. The runtime team agreed in February 2025, and the fix merged on 24 October 2025 under milestone 11.0.0. It was not backported: release/11.0 creates both instruments with CreateObservableUpDownCounter, and release/10.0 still creates them with CreateObservableCounter.

What a one-second interval cannot resolve

Choosing the provider that reports a level fixes the sign and not the sampling. ThreadPool Queue Length is one instantaneous reading per interval, and the interval is a long time. A second workload queues 500 items at the start of every second, each spinning 5 ms and allocating 64 KiB, which fills the queue to roughly 500 and drains it in about 200 ms. Sampled every 10 ms, one cycle looks like this:

  t (ms)   queue
       0     474
      88     250
     179      32
     267       0
     354       0
     442       0
     529       0
     616       0
     704       0
     792       0
     880       0

Across the run the queue reached 497 at its highest and was non-empty for 18 per cent of the samples. ThreadPool Queue Length reported 0 in all twenty-three readings, in five consecutive fresh processes — one hundred and fifteen readings, none of them non-zero. That is not bad luck. The burst period and the poll period are both one second, so a sample that lands in the idle gap once lands there every time. Changing only the burst period to 700 ms, leaving the amplitude and the interval alone, turned the same column into 81, 363, 199, 452, 9, 242 and 487 among the zeros. The counter did not improve; the workload stopped hiding behind it.

Interval totals survive what spot samples do not. Over one of those runs ThreadPool Completed Work Item Count summed to exactly 10,000, the number of items the harness queued, and Allocation Rate summed to 656,471,696 bytes against the 656,873,304 the process reported at exit, short by the part of the last second the collector never sampled. What the interval removes is shape rather than quantity: the allocation rate read 32.8 MB/s every second, against roughly 152 MB/s sustained inside the 215 ms window that produced it, a factor of 4.6.

What this changes

Read the Counter Type column before the value. Metric is a level, Rate is a per-interval figure, and on .NET 10 a Rate row whose name describes a level — a queue length, a thread count — is a difference rather than a measurement. Every genuinely cumulative row is exact, in either provider: dotnet.thread_pool.work_item.count reported 500 per second and summed to the 10,000 items its process completed, and dotnet.gc.heap.total_allocated summed to 656,559,712 bytes against the 657,620,256 that process reported at exit.

For a queue depth on .NET 10, type the prefix: --counters 'EventCounters\System.Runtime[threadpool-queue-length]'. It cannot be combined with the unprefixed provider, so a session that needs both instrument sets needs two sessions.

--refresh-interval is parsed as System.Int32, so one second is the shortest interval that can be requested and 0.1 is rejected outright. Passing 0 does two different things: on the Meter path AggregationManager.MinCollectionTimeSecs is 0.1 and the request is rounded up to it, giving ten readings per second; on the EventCounters path CounterGroup reads an interval of zero or less as a signal to turn counters off, and the collector writes a CSV containing nothing but its header row.

And the meaning of the row changes on upgrade rather than on redeployment. On the .NET 11 release branch the same counter name is published as an up-down counter, so a panel that has been quietly plotting deltas will start plotting depths, and a threshold tuned against the old behaviour will fire on the first busy second. Both numbers are real. Only one of them is the queue.

---

Measured on .NET 10.0.12, SDK 10.0.401, macOS 26.6.2 arm64, 12 cores, Release configuration, dotnet-counters 10.0.745401. Ground truth is ThreadPool.PendingWorkItemCount read from a dedicated foreground thread every 10 ms; the tables are printed by a script that joins that sampler’s CSV with the collector’s CSV on the wall-clock second and are pasted as printed. The held-queue table is one process with two collector sessions attached at once; the burst figures are single-session runs, because a second session lengthened the drain from a median of 215 ms to 228 ms with a 464 ms outlier and the queue row stopped reading zero. A spot sample cannot report a maximum, so the true-queue column is the range the 10 ms sampler saw within that second and not a mean. The registration quoted is from RuntimeMetrics.cs on release/10.0, read on 13 September 2026 alongside release/11.0; the interval constants are from AggregationManager and CounterGroup in the same branch.