In the previous post I described how a failed complete after a scheduled retry leaves two live copies of one message. That needed a trigger: a lock expiring at the wrong moment. This post covers two things that produced that trigger and then made it worse.
- Prefetch starts the lock clock before your handler runs.
- My backoff delay was measured from the wrong clock, so “retry in 5 minutes” often meant “retry right now”.
Setup recap
3 pods × 5 concurrent listeners, receiver prefetch of 5, queue lock of 60 seconds, a downstream call averaging 16 seconds (p99 43 s, hard timeout 120 s), and a lock renewal every 30 seconds while the call is in flight.
I had already added that 30-second renewal after an earlier test. I thought lock expiry was handled.
Prefetch and the lock clock
With prefetch, the client fetches messages ahead of demand into a local buffer. In peek-lock mode every prefetched message is already locked, and its lock clock starts when it reaches the buffer, not when your handler starts.
My renewal logic starts when processing starts. So it can’t protect a message that is sitting in the buffer. The SDK’s own auto-renewal behaves the same way: it renews for up to five minutes per message, but only from the moment the message is handed to the application. I confirmed that on a dev namespace. With processing held 400 s after handover, every lock expired about 5 min 12 s after handover.
That’s harmless when processing is fast. It’s a trap when a slot can stall.
How the trap fired
Just before the outage, the downstream system stopped answering instead of refusing connections:
| Calls started | Count | Succeeded | Longest | Median |
|---|---|---|---|---|
| T+0:00 to 0:29 | 14 | 9 | 120.1 s | 22.2 s |
| T+0:30 to 0:59 | 7 | 0 | 120.0 s | 120.0 s |
| T+1:00 to 1:59 | 0 | 0 | ||
| T+2:00 to 2:29 | 67 | 0 | 0.2 s | 0.1 s |
For roughly two minutes, every consumer slot was holding a hung call, so nothing new started. The messages already prefetched into each pod’s buffer sat there, locked, past their 60-second lock. When the hung calls finally timed out at 120 s, the slots pulled those stale messages from the buffer. They were processed (and failed fast against the dead downstream), and then every settlement failed with The lock supplied is invalid.
That’s the trigger for the duplicate-copy bug.
The same mechanism on the success path
Renewal protects a message once processing begins, but not before. During the test, 125 completes failed with an expired lock after a perfectly successful downstream call, which led to at least 116 duplicate submissions. Same root cause, different symptom.
What I’d change (not yet validated)
I want to flag that these are recommendations from the analysis. I have not yet re-run the load test with them.
- Set prefetch to 0 or 1 instead of equal to concurrency. Messages are then locked only when a slot is free. The extra fetch round trip is small next to a 16-second call. The cost of prefetch is real here and the benefit is not.
- Raise the queue lock duration from 1 minute to 5 minutes (the maximum). It widens the window a stalled slot can absorb. It does not remove the problem; it moves the cliff.
I’d also put the thing the SDK can’t do on my checklist: a buffer-wait time metric, the gap between a message’s lock acquisition and the handler start. It would have shown this immediately.
The wrong clock
The second defect was embarrassing in hindsight. The retry delay was computed like this:
// Before
scheduleMessage(copy, message.getEnqueuedTime().plus(delay))
getEnqueuedTime() is when the message entered the queue, not when processing failed. Under a backlog that’s wrong by hours. The queue drained at about 1,850 orders per hour, so by the time the front of the queue was processed, messages had been waiting about 2 hours 40 minutes.
For those messages:
intended: failure time + 5 min
computed: enqueue time (≈ 2 h 40 m ago) + 5 min = already in the past
result: the scheduled copy is active immediately
Service Bus accepts a past scheduled time and makes the message visible right away. No error, no warning.
Why mostly the first retry was hit
For the 2nd and 3rd retries, the message being processed is itself a copy, and a copy’s enqueue time is when it became active, which is recent. So the first retry was nearly instant, and the later ones landed at closer to the intended delay. That’s why 1,482 orders failed twice during the five-minute outage, but only 4 failed a third time.
The evidence
The designed gap between an order’s first and second failure was at least 300 seconds. What I measured, across 1,259 orders where both failures were logged:
| min | p10 | p50 | p90 | max |
|---|---|---|---|---|
| 2 s | 144 s | 150 s | 156 s | 171 s |
Not one order reached 300 seconds. The 150 s figure isn’t a delay at all. It’s the time the fast-failing consumers needed to make one pass through the backlog (about 1,900 messages at about 700 per minute). The copies were active immediately and simply waited their turn at the back of the queue. The 2 s minimum was a redelivered original from the lock-loss bug.
The dead-letter queue even preserved the proof: copies created at 22:46 with a scheduled time of 20:10, more than two hours earlier.
The fix is one line
// Measured from now, not the message's enqueue time. Under a backlog the
// enqueue time can be hours old, which puts every backoff in the past.
OffsetDateTime scheduledTime = OffsetDateTime.now().plus(delay);
A different code path in the same listener already used now(). The two paths had different semantics for the same concept, which is how this survived review.
Retry budget math
With a limit of 3 attempts and backoffs of 5/10/20 minutes, an order should be tried at roughly t, t + 5 min and t + 15 min. The third failure schedules a copy for t + 35 min that is dead-lettered on arrival. So an order should only die if the dependency stays down for about 15 minutes.
What actually happened: attempts at t, t + 2.5 min and t + 10 min. The budget was cut by a third, and the margin that made the design safe was gone. A roughly 10-minute outage would have dead-lettered about 1,500 orders. This one lasted five.
How the three defects combined, for an order whose lock expired during the outage:
| Defects present | Outcome |
|---|---|
| Wrong clock only | Failed twice during the outage, succeeded after. Within budget, no margin |
| Lock loss + non-atomic retry, correct clock | Two live instances, each retrying 5+ minutes later; both succeed afterwards; the downstream is called twice |
| All three | Both instances retried immediately, used the shared counter up during the outage, dead-lettered |
| With the fixes | One live instance; retries land after the outage; one order created |
What I wish Service Bus had
Everything in this post, and the previous one, exists because there is no “put this back in N minutes” operation on a received message. Abandon has no delay. Defer needs bookkeeping. Scheduling a copy resets the delivery count, moves the message to the back of the queue, and requires you to atomically manage the original.
If the broker offered an “abandon with delay” or per-message redelivery backoff, the copy, the shared counter, the transaction and the stale clock would all go away. Until then, schedule-and-complete in a transaction, with the delay computed from now(), is the least-bad pattern I know. And it still has a surprise: what “5 minutes” means once the queue is deep. That’s the next post.
Lessons
- A lock’s clock starts when the broker hands over the message, not when your code does. Anything between those two moments, a prefetch buffer included, is time you can’t renew.
- Prefetch trades lock safety for throughput. Check that you’re actually getting the throughput. If your handler takes 16 seconds, you aren’t.
- Time arithmetic on a message property is a bug waiting for a backlog. Compute delays from
now(). - Test with a deep queue. Both defects were invisible with a small backlog and only showed up with hours of queue age.