Diagnosing ASP.NET Core Thread-Pool Starvation on IIS with Counters and a Load Test

Requests to an ASP.NET Core app on IIS start taking 3 to 15 seconds while CPU sits at 20 percent. Thread count is high, throughput in Microsoft.AspNetCore.Server.Kestrel request counters collapses, and the app recovers by itself after a few minutes of quiet. Nothing appears in the event log, no 500s are returned, and a restart clears it for an hour. That pattern is thread-pool starvation: every worker is blocked waiting on something, so newly injected threads cannot pick up queued work fast enough.

The reason it hides so well is that starvation looks like slowness, not failure. Latency rises, requests still complete, and the machine reports idle CPU. Capacity planning based on CPU graphs will never surface it; you need the counters that describe queue depth and thread injection, and a load profile that reproduces the block. For hosts where the app and IIS share a small core count, look at Windows VPS configurations built for ASP.NET Core workloads before adding threads as a fix.

The signature in performance counters

Add these counters under Process for the w3wp instance: % Processor Time, Thread Count, Private Bytes. Under ASP.NET Core Requests or the Kestrel counters exposed via EventSource, watch request rate and average duration. Under .NET CLR LocksAndThreads, watch # of current logical Threads and Contention Rate / sec. A thread count pinned near the configured minimum with a flat request rate is the classic shape.

The clearest single signal is that thread count sits at the minimum worker thread level, usually the core count, and does not rise despite a full request queue. The runtime injects roughly one thread per 500 ms after that, so a workload that queues 200 blocking operations takes over a minute to drain even after the blocking call returns. During that minute, every request that arrives waits behind work that has already finished.

Find the synchronous block, not the thread count

Raising thread minimums treats the symptom. The cause is a blocking call on a request thread: .Result or .Wait() on a task, synchronous database calls in an async controller, File.ReadAllText on a network share, a lock held across an await, or a synchronous HTTP client call. Search the codebase for .Result;, .Wait(), GetAwaiter().GetResult(), and Thread.Sleep in request paths.

Under a repro, capture a stack dump with dotnet-stack report --process-id <pid> or a full dump through Task Manager’s create dump file option, then inspect with dotnet-dump analyze. Count how many threads sit inside the same frame; ten or more worker threads at one call site names the culprit immediately. Two dumps thirty seconds apart separate transient waits from a genuine deadlock.

What the autoscaling of threads really costs

The CLR grows the worker pool conservatively on purpose. Unbounded growth would let a burst of blocking calls consume a gigabyte of stack space and thrash the scheduler. When you call ThreadPool.SetMinThreads(workerThreads, completionPortThreads) you shorten that ramp, at the cost of memory and context switching. Set it once at startup, before the first request, in Program.cs, not from a controller where it changes behaviour mid-flight.

A reasonable starting point for an API that must survive a known burst is 2 to 4 times the core count, paired with a matching ThreadPool.SetMaxThreads ceiling so a runaway does not spiral. On a 4-vCPU instance, 16 minimum worker threads is defensible; 200 is a sign that the real problem was never addressed and the app will fall over at a lower concurrency than before.

IIS settings that change the math

In-process hosting under the ASP.NET Core Module runs inside w3wp, so the IIS worker process thread pool and the app share a budget. Out-of-process hosting puts Kestrel in dotnet.exe behind a proxy hop, which separates the pools but adds a serialization step. If a slow non-async module or an ISAPI filter runs in the same worker, both hosts suffer it. Check the app pool for Maximum Worker Processes above 1 on a small box as well; web gardens multiply memory and thread pressure.

processModel request queue limits matter too. When the queue fills, IIS returns 503 instead of slow responses, so 503 waves mixed with latency spikes point at the same saturation. Raising requestQueueLimit buys time but not capacity.

Build a load test that reproduces it

Use a tool that keeps a fixed concurrency level and reports percentiles: bombardier -c 64 -d 120s http://127.0.0.1:5000/api/orders, or wrk with a Lua script for a POST body. Run it against the local Kestrel port first, then through IIS, so the proxy hop is ruled out. Record p50, p95, p99 and requests per second at concurrency 8, 32, and 128.

Starvation shows as p50 that stays flat while p99 climbs by an order of magnitude with rising concurrency, then a hard throughput plateau well below what the CPU could deliver. A healthy async app scales until CPU saturates; a starved one plateaus at roughly (threads / blocking time) requests per second. Watch the perfmon counters live during the run and note the thread count at the plateau.

Fix the code, then tune the floor

Convert the blocking paths to async all the way down, using await and ConfigureAwait(false) in library code. Give the database context an async command timeout you actually enforce and a connection pool size that matches worker concurrency, otherwise async code collapses back into waiting on a connection. Replace per-request locks on hot caches with immutable snapshots or SemaphoreSlim awaited asynchronously.

Then revisit the minimums and re-run the same test. The threads-per-second slope in the counters tells you whether the change worked: with blocking removed, thread count should barely move at concurrency 128. If it still climbs, an async-over-sync wrapper somewhere is presenting an async signature over a blocking call, and the dump will show the same frame as before.

What to alert on in production

Alert on the shape rather than a single sample: thread count held above 80 percent of the maximum for two minutes, or p95 latency above threshold while CPU is under 40 percent. Both conditions are cheap to compute and both fire long before user-visible timeouts. Publish the counters from the app itself with EventCounters to Application Insights or a metrics endpoint so you are not dependent on a perfmon session being open.

Record the fix and its measured effect in the runbook next to the pool settings, because the next person to see slow responses will otherwise raise the minimums again. Combined with a host that has headroom for the extra threads, the change holds. If your current instance runs two vCPUs for an API with database round trips, step up to a Windows VPS with more cores and memory and re-measure before adding any tuning.

Leave a Comment