why your api slows down before cpu hits 100%
your API can start timing out while the CPU dashboard still looks fine.
in a typical Node.js process, application JavaScript runs on one event loop thread. giving that process eight cores does not automatically parallelize parsing, pricing calculations or serialization. worker threads or multiple processes are needed to distribute that JavaScript work.
if a dashboard averages CPU usage across eight cores, one fully busy core contributes roughly 12.5% of total capacity. the event loop can therefore be saturated while the machine-wide average remains low.
that number depends on the metric's denominator. if the process is limited to one core and usage is shown relative to that limit, consuming one core corresponds to 100%, not 12.5%. check whether the dashboard reports usage against machine capacity, a container limit or a CPU request.
requests can also wait for a database connection, a lock or a remote service without consuming much local CPU.
look at the resource they are waiting for.
a useful starting point is little's law:
l = lambda * w
in a stable system, l is the average number of requests inside it, lambda is throughput, and w is average time inside it, including waiting.
at 10,000 requests/s and 200 ms average latency, that means 2,000 requests in flight. if latency reaches 2 seconds at the same throughput, it becomes 20,000.
to understand how waiting grows, take an M/M/1 model: poisson arrivals, independent exponential service times, one server and an unlimited queue.
with arrival rate lambda below service rate mu:
w = 1 / (mu - lambda)
for mu = 1,000 requests/s, average service time is 1 ms. at 500 requests/s, average response time is 2 ms. at 900, it is 10 ms. at 990, it is 100 ms.
these are model calculations. real APIs have several queues, bursts and different request costs. the formula describes that specific model, not a universal API latency curve.
the practical problem is that waiting requests still retain resources.
each can hold request context, buffers, promises, timers and an open connection. rising concurrency increases memory pressure. garbage collection and timeout handling can then consume capacity that was previously available for useful work.
async code makes it easy to create this backlog.
in Node.js, awaiting a network operation releases the event loop. parsing its response and recomputing a price still run synchronously unless explicitly moved elsewhere. when many operations finish together, their callbacks compete for the same thread.
starting 10,000 promises can also leave most of them waiting for a much smaller database connection pool. the queue has moved into the client library.
measure queue wait separately from execution time. include connection acquisition, event loop delay, per-core CPU, active requests and p99 latency. a handler span that begins after admission can miss the delay the client actually experienced.
then bound concurrency at the resource that needs protection.
in Rust, a semaphore can cap active operations. a bounded channel can cap queued work. thousands of tasks awaiting a semaphore still form a waiting population, so admission itself needs a limit or a deadline.
the same applies to Node.js concurrency limiters: check whether their pending queue is bounded.
give each request an end-to-end deadline. time spent waiting consumes that budget. check it before expensive work starts, propagate the remaining budget downstream, and cancel work where supported when it is no longer useful.
when the queue is full, backpressure only works if the producer actually slows down. otherwise reject excess work early, with 429 for rate limits or 503 for temporary service overload.
retries need their own budget. if every request makes three attempts, 10,000 original requests can generate 30,000 attempts. a timeout does not prove that the previous attempt stopped running. bounded retries, backoff and jitter help prevent clients from synchronizing another burst.
for a price stream, the queue policy can depend on the data. pending updates for the same token may be coalesced when consumers only need the latest state. transaction events need different handling because intermediate events can matter.
autoscaling helps when more replicas add capacity at the actual bottleneck. it takes time to start them and warm their caches. if the database is already saturated, more API pods may only create more competing queries.
under overload, measure how many requests finish successfully within their deadline. a larger queue can make the service accept more work while delivering fewer responses that arrive in time.
background jobs may tolerate an hour of backlog. a trading API needs an explicit limit on how long a request is allowed to wait.
queueing model reference: https://www.cl.cam.ac.uk/teaching/0304/CompSysMod/CSMSlides.pdf
event loop reference: https://nodejs.org/en/learn/asynchronous-work/dont-block-the-event-loop