During a 10,000-order performance test, a downstream system went unreachable for about five minutes. The consumer was designed to ride that out: three attempts per order, spread over roughly 15 minutes. Instead, five orders burned all three attempts and landed in the dead-letter queue, and at least 116 orders were submitted to the downstream system twice.
Nothing was “broken” in the usual sense. No exception pointed at the cause. The queue, the SDK and the app each did exactly what they were told. The bug lived in the gap between two operations that I had assumed were one.
This is the first of three posts about that test. This one covers the duplicate-message mechanism and the fix. The next one covers prefetch and the missing “retry later” primitive, and the last one covers throughput.
The setup
- Azure Service Bus Premium, one queue, peek-lock mode.
- Lock duration 60 seconds. Max delivery count 10. No sessions, no duplicate detection.
- 3 pods × 5 concurrent listeners = 15 messages in flight.
- Per message: read a retry counter from the database, call a SOAP endpoint on the downstream system, then settle the message based on the outcome.
- The listener renews the lock every 30 seconds while the downstream call runs.
Settlement rules:
| Outcome | Action |
|---|---|
| Success, business error | complete |
| Retryable error (connect failure, timeout, reset) | increment retry counter, schedule a delayed copy, complete the original |
| Retry counter at max | dead-letter |
Peek-lock in one paragraph
The receiver gets a message plus an exclusive lock for the queue’s lock duration. You must settle it (complete, abandon, dead-letter, defer) while you hold the lock. If the lock expires, the broker makes the message available again immediately, and settling with the old lock fails with The lock supplied is invalid. An expired lock cannot be renewed.
That last sentence is the whole story. Everything below is a consequence of it.
Service Bus can’t delay a received message
You’d think “retry this in five minutes” would be a first-class operation. It isn’t:
| Option | Behavior | Usable as delayed retry? |
|---|---|---|
| Abandon | Available again immediately | No delay |
| Defer | Set aside; fetch later by sequence number | No timer, you build the bookkeeping |
| Let the lock expire | Same as abandon, after the lock duration | Delay is fixed at 60 s |
| Schedule a copy, complete the original | Copy becomes visible at a time you choose | Yes |
So the standard pattern is: send a brand-new message with a ScheduledEnqueueTime, then complete the original. Two operations:
// Before: two independent operations
return sender.scheduleMessage(copy, scheduledTime) // step 1: the copy is now live
.then(receiver.complete(original)); // step 2: may fail
Step 1 commits on its own. If step 2 fails, step 1 is not undone.
The failure matrix
| Schedule | Complete | Live instances afterwards |
|---|---|---|
| ok | ok | 1 (the copy) |
| fails | not attempted | 1 (the original, redelivered after lock expiry) |
| ok | fails (lock expired) | 2: the copy and the redelivered original |
The third row is the bug. The lock had expired, so complete was rejected. The copy was already scheduled, and the broker redelivered the original because nobody settled it. Now two independent instances of the same order exist.
It gets worse, because a scheduled copy is a brand-new message. The broker’s delivery count restarts at 0 on every copy, so you can’t use it to cap retries. I tracked attempts in a database column keyed by order. Both instances read and incremented that same counter:
- The order’s three attempts were now shared by two instances, so it burned them twice as fast.
- Read-then-increment wasn’t atomic. Two instances processing at the same moment both read
2, both failed, both scheduled another retry, and pushed the counter to4. Both stayed alive. - Every retry spawned another copy per instance, so the duplication never collapsed.
What it looked like in production
Five reschedules failed with lock supplied is invalid within a four-second window. Those were exactly the five orders that were later dead-lettered, and each had two copies in the DLQ. About three minutes later, with the downstream still down, both copies of each order ran:
- Three orders: one copy read counter
2and retried, the other read3and dead-lettered. - Two orders: both copies retried and pushed the counter to
4.
A sequence view of one order:
original → downstream call fails counter 0 → 1
original → schedule copy A (live now)
original → complete FAILS (lock expired)
original → redelivered 2 s later, fails counter 1 → 2
original → schedule copy B, complete OK
A and B → both run ~3 min later, both read 2, both fail counter → 4
A and B → each schedules a 20-minute retry
... → both reach the front of the queue, counter ≥ 3, dead-lettered
The fix: make it one operation
Service Bus supports transactions. A transaction can include a send (including a scheduled send) and a complete of the message you received. The broker applies them together or not at all.
private Mono<Void> scheduleCopyAndComplete(
ServiceBusReceivedMessage original, ServiceBusMessage copy, OffsetDateTime when,
ServiceBusReceiverAsyncClient receiver, ServiceBusSenderAsyncClient sender) {
return Mono.usingWhen(
sender.createTransaction(), // begin
tx -> sender.scheduleMessage(copy, when, tx) // op 1
.then(Mono.defer(() -> receiver.complete(original,
new CompleteOptions().setTransactionContext(tx)))), // op 2
sender::commitTransaction, // both ok
(tx, err) -> sender.rollbackTransaction(tx), // either failed
sender::rollbackTransaction); // cancelled
}
Mono.usingWhen ties the transaction’s fate to the outcome of the work inside it: commit on success, roll back on error or cancellation.
| Schedule | Complete | Commit | Result | Live instances |
|---|---|---|---|---|
| ok | ok | ok | Copy visible at the scheduled time; original removed | 1 |
| ok | fails (lock expired) | rolled back | Copy discarded; original redelivered | 1 |
| fails | not attempted | rolled back | Original redelivered | 1 |
| ok | ok | fails or times out | Broker aborts; original redelivered | 1 |
Every row ends at exactly one live instance. That’s the property I actually wanted.
The constraints nobody mentions
I hit most of these the hard way.
- A transaction is bound to one AMQP connection. In the Java SDK, clients share a connection only if they’re built from the same
ServiceBusClientBuilderinstance. My sender and receiver each created their own builder, so they couldn’t participate in one transaction. Fix: one shared builder, and rebuild the receiver from that same builder after a failure. - Not available on the Basic tier. Standard and Premium only. On Basic,
createTransactionfails, which would have made every retry fail. Check every environment before you ship. - One entity, unless you enable cross-entity transactions. Receiving from and scheduling to the same queue stays within one entity, so this was fine. A dead-letter replay that spans the DLQ and the main queue is a different story.
- Transactions time out after two minutes. Mine wraps two quick operations. The slow downstream call happens before the transaction begins. Don’t put your slow work inside it.
- Cost: two extra round trips (begin and commit), paid only on the retry path. Negligible next to a 16-second downstream call.
Checking it against a real broker
Unit tests proved the wiring: same transaction context on both operations, commit on success, rollback on a failed complete or failed schedule. But a mock can’t prove what the broker does, so I ran it against a real namespace with the downstream simulated. I held each message 400 seconds after delivery so every lock expired before the call, with the downstream returning connection resets throughout.
| Time | Without the fix | With the fix |
|---|---|---|
| 6 min 40 s | Attempt 1 fails; copy live immediately; complete fails. 2 live | Attempt 1 fails; broker rejects the transaction with amqp:transaction:rollback, copy discarded. 1 live |
| 11 min 52 s | Attempt 2 by redelivered original. 3 live | Attempt 2, rolled back again. 1 live |
| 13 min 21 s | Attempt 3 by first copy. 4 live | |
| End of watch | Retries exhausted, 4 live | 2 downstream calls, 1 live, 0 scheduled |
Without the transaction, all three attempts were spent in under seven minutes. With it, the broker’s rollback discarded every copy.
What this does not fix
I want to be precise here, because this is where blog posts usually overclaim.
- A lost lock still costs an attempt. The redelivered original is processed and counts against the retry budget. The transaction prevents the second instance, not the redelivery.
- The success path is untouched. During the same test, 125
completecalls failed with an expired lock after a successful downstream call. Service Bus redelivered those, and at least 116 orders were submitted twice. The downstream system answeredDUPLICATE_ORDER, which my code records as success, so no order was lost, but the downstream did duplicate work. A transaction doesn’t help: there’s no second operation to make atomic. The real fix is preventing the lock loss (the next post) and idempotency downstream. - The counter is still read-then-increment. With a single live instance it stops mattering in practice, but it should be one atomic database update.
- Validated small. The real-broker runs used two to ten orders on one pod. I haven’t yet re-run the full outage under load.
Lessons
- “Do A, then B” is a distributed-systems smell wherever A is visible to other consumers before B completes. If B can fail, A needs to be undoable.
- Peek-lock settlement can fail on a message you handled perfectly. Treat
completeas fallible, not as a formality. - Check for the same pattern elsewhere. In the same codebase, other listeners did
scheduleMessage(...).subscribe()fire-and-forget followed by a complete. That’s worse: the two run concurrently, so a failed schedule plus a successful complete loses the message with only an error log to show for it. Chain the operations at minimum; use a transaction where you can. - Make the downstream idempotent. Everything I did reduces duplicates. Only idempotency makes them harmless.