This is the third post from the same performance test. The first was about duplicate messages and the second about prefetch and backoff clocks. This one is the simplest and probably the most useful: what actually determines how fast a queue drains.
The result
I loaded about 10,000 orders onto a queue. The consumer, which reads a message, does a couple of database operations and posts to a downstream SOAP endpoint, is cheap. Nothing in my own code was slow.
The queue took 5 hours 13 minutes to drain 8,303 messages through that path. That’s roughly 31 per minute, or about 1,850 per hour.
Little’s law, not mysticism
Throughput ≈ concurrency ÷ latency.
| Input | Value |
|---|---|
| Concurrent calls (3 pods × 5 listeners) | 15 |
| Downstream latency, p50 | 16.2 s |
| p90 / p99 / max | 33.1 s / 43.3 s / 120.1 s (timeout) |
| Predicted throughput at the median | 15 ÷ 16.2 s ≈ 55/min |
| Observed | ≈ 31/min |
The observed rate is below the median-based prediction because the mean is dragged up by the tail, and a slot stuck on a 120-second timeout produces nothing for two minutes. A queue consumer’s real service time is the average of that distribution, not the median.
The point is that the speed of my listener never appears in this equation. I could have made it ten times faster and the drain would still have taken five hours. Throughput was set by:
- How many calls I allowed in parallel, and
- How long the dependency took per call.
If you’re asked “how long will it take to drain N messages?”, the answer is N × (average downstream latency) ÷ concurrency. Measure the dependency under load, not in isolation. Mine was much slower with 15 callers than a single call suggested.
The good news: the cap protected the dependency
Concurrency was deliberately limited: 5 listeners per pod, prefetch equal to concurrency. At no point could the system have more than 15 calls in flight against the downstream.
That mattered more than it looks. A message queue’s best feature is that it converts a burst into a steady rate. If I had let the consumers scale to, say, 100 concurrent calls, a slow downstream would have slowed down further (or tipped over), the 120-second timeouts would have multiplied, and the retry traffic would have made it worse.
Instead, every message was eventually consumed. Slow, but nothing was dropped and the dependency was never bombarded. For an order pipeline, that’s the right trade. If the business needs it faster, the lever is a faster downstream (or a deliberate, tested increase in concurrency), not a bigger consumer fleet.
A note on the knob itself: it’s tempting to raise concurrency until the queue drains. The downstream’s tail latency is what you’re spending when you do. I’d rather change it with a number in hand (p99 at 15 vs 30 callers) than by feel.
“Retry in 5 minutes” is a floor, not a delay
When the downstream failed, my code scheduled a copy of the message for five minutes later. In an empty queue that means roughly five minutes. In a deep queue it doesn’t.
A scheduled copy becomes a regular message at its scheduled time and joins the back of the queue. Whatever is ahead of it must be processed first. So:
real delay = max(scheduled delay, time to drain everything ahead of it)
With a queue that takes hours to drain, my “retry in 5 minutes” messages were picked up closer to an hour later. Here is the shape of it from the incident: twenty-minute copies scheduled at 23:06 weren’t reached until 00:15 — about 70 minutes later — because the queue was still draining thousands of messages ahead of them. By the time they were handled the retry counter had already used up its budget, so they were dead-lettered on arrival.
Two practical consequences:
- A retry’s real latency is unbounded under backlog. Don’t design an SLA around the scheduled delay.
- Retry budgets assume that the delay is the delay. I’d computed “dead only if the dependency stays down 15 minutes”. With a deep queue, the time between attempts was dominated by backlog, and an attempt could land long after the dependency recovered, or, when mixed with the clock bug from the previous post, much too early.
The mirror-image problem existed during the outage itself. When the downstream failed fast (about 0.1 s per call), the consumers raced through the backlog at roughly 700 messages per minute, producing 3,368 retryable failures in under five minutes and scheduling a copy for each. The rate that was 31/min against a healthy downstream became 700/min against a dead one. A fast-failing dependency spends your retry budget far faster than a slow one. A circuit breaker, or pausing the consumer when the dependency is clearly down, would have turned that into near-zero wasted attempts. I didn’t have one.
What I’d do about it
None of these are exotic:
- Size the system by the dependency. Know the downstream’s throughput at your concurrency, and give it a number: “this queue drains N/hour”. Alert on queue age, not just depth.
- Keep concurrency capped, and change it deliberately. The cap was why the downstream survived.
- Treat scheduled retry as “not before”, never “at”. If timing matters, measure the retry’s actual delay in production.
- Add a circuit breaker. When the downstream is failing fast, stop pulling messages (or back off the whole consumer) rather than burning each message’s attempts.
- Compute backoff from
now(), and prefer fewer long-lived messages in flight over many short-lived retries. - Make the downstream idempotent. The duplicate submissions across this series were survivable only because it returned
DUPLICATE_ORDERinstead of creating a second order.
Series summary
| Post | Problem | Fix |
|---|---|---|
| Lock expiry duplicates | Schedule-then-complete isn’t atomic; lock loss leaves two live copies | Schedule and complete in one Service Bus transaction |
| Prefetch and backoff | Prefetched messages lose their lock before handling; backoff computed from enqueue time | Prefetch 0/1, longer lock, delay from now() |
| This post | Throughput is bound by the slowest dependency; a delayed retry rejoins the back of a deep queue | Capacity-plan by the dependency, circuit-break on fast failures |
The unifying lesson: a queue decouples producers from consumers, but it doesn’t decouple you from your slowest dependency. It only controls how you wait for it.