A queue is a loan, not a fix

Two in the morning, and the service is falling behind. Requests are timing out, the error rate is climbing, and someone in the incident channel types the sentence that feels like relief: “let’s just put a queue in front of it.” You do. The timeouts stop. The errors clear. Everyone exhales and goes back to bed. And for a while — sometimes a long while — it looks like the queue fixed it. It didn’t. It borrowed against it, and the loan comes due later, with interest, usually at a worse hour.

Here’s what a queue actually does, precisely, because the precision is the whole point. A queue absorbs bursts. If work arrives in uneven spikes but your workers can handle the long-run average, a buffer in the middle smooths the spikes into something steady, and that’s genuinely valuable — it decouples the thing producing work from the thing doing it, so a momentary flood doesn’t knock anyone over. What a queue cannot do is add capacity. It does not make your workers faster. If work arrives, on average and over time, faster than you can finish it, the queue does not save you. It just changes the shape of the failure — from loud and immediate to quiet and delayed.

There’s a plain law underneath this, and it’s worth carrying around. On average, the number of items sitting in a queue equals how fast they arrive multiplied by how long each one waits. (That’s Little’s Law, and it holds remarkably generally.) Read it the direction that stings: if items keep arriving faster than you drain them, the queue length grows, and because length and wait are tied together, the wait grows right along with it. So the request that used to fail in two seconds now succeeds — after ninety. By the time your worker picks up a job, the user who triggered it has refreshed, given up, or triggered three more. An unbounded queue quietly converts a capacity problem into a latency problem, and then a latency problem into a serving-stale-work problem, and the whole time your dashboards look fine because nothing is technically erroring.

The honest alternative has an unglamorous name: backpressure. Instead of silently swallowing more work than you can handle, you push back — you signal up the chain that you’re full, and let the pressure propagate toward the source. Concretely that means bounding your queues (a queue with no maximum is a memory leak with good PR), and deciding in advance what happens when a bounded queue fills: reject the newest work, drop the oldest, sample, or degrade to a cheaper response. That decision is not a detail. It’s the moment you choose, deliberately and while calm, how your system behaves at its worst — rather than discovering the default behavior live, at two in the morning, when the default is “fall over.”

The turn is that backpressure isn’t free either, and pretending it is just relocates the pain. When you push back, the pressure has to land somewhere — often on a user, as a “we’re busy, try again,” or on an upstream service that now has to hold or shed the work itself. That’s a real product decision wearing an infrastructure costume: is it kinder to reject a request cleanly and immediately, or to accept it and answer slowly, or to drop the oldest thing in line so the freshest gets through? There’s no universally right answer. A payment probably deserves a clean rejection over a stale success; a live metrics feed probably wants the newest reading even if that means discarding the last one unshown. The point isn’t that backpressure is always correct. It’s that someone is going to absorb the overload, and backpressure is you deciding who and how, on purpose, instead of letting an invisible growing buffer decide it for you later.

So the next time the incident channel reaches for a queue, ask the question the queue is designed to let you avoid: are we buffering a burst, or are we hiding the fact that we ran out of capacity? A buffer for a burst is good engineering. A buffer for a shortfall is a countdown you can’t see.