During a 10,000-order performance test, a downstream system went unreachable for about five minutes. The consumer was designed to ride that out: three attempts per order, spread over roughly 15 minutes. Instead, five orders burned all three attempts and landed in the dead-letter queue, and at least 116 orders were submitted to the downstream system twice.

Nothing was “broken” in the usual sense. No exception pointed at the cause. The queue, the SDK and the app each did exactly what they were told. The bug lived in the gap between two operations that I had assumed were one.

This is the first of three posts about that test. This one covers the duplicate-message mechanism and the fix. The next one covers prefetch and the missing “retry later” primitive, and the last one covers throughput.

The setup

  • Azure Service Bus Premium, one queue, peek-lock mode.
  • Lock duration 60 seconds. Max delivery count 10. No sessions, no duplicate detection.
  • 3 pods × 5 concurrent listeners = 15 messages in flight.
  • Per message: read a retry counter from the database, call a SOAP endpoint on the downstream system, then settle the message based on the outcome.
  • The listener renews the lock every 30 seconds while the downstream call runs.

Settlement rules:

OutcomeAction
Success, business errorcomplete
Retryable error (connect failure, timeout, reset)increment retry counter, schedule a delayed copy, complete the original
Retry counter at maxdead-letter

Peek-lock in one paragraph

The receiver gets a message plus an exclusive lock for the queue’s lock duration. You must settle it (complete, abandon, dead-letter, defer) while you hold the lock. If the lock expires, the broker makes the message available again immediately, and settling with the old lock fails with The lock supplied is invalid. An expired lock cannot be renewed.

That last sentence is the whole story. Everything below is a consequence of it.

Service Bus can’t delay a received message

You’d think “retry this in five minutes” would be a first-class operation. It isn’t:

OptionBehaviorUsable as delayed retry?
AbandonAvailable again immediatelyNo delay
DeferSet aside; fetch later by sequence numberNo timer, you build the bookkeeping
Let the lock expireSame as abandon, after the lock durationDelay is fixed at 60 s
Schedule a copy, complete the originalCopy becomes visible at a time you chooseYes

So the standard pattern is: send a brand-new message with a ScheduledEnqueueTime, then complete the original. Two operations:

// Before: two independent operations
return sender.scheduleMessage(copy, scheduledTime)   // step 1: the copy is now live
    .then(receiver.complete(original));              // step 2: may fail

Step 1 commits on its own. If step 2 fails, step 1 is not undone.

The failure matrix

ScheduleCompleteLive instances afterwards
okok1 (the copy)
failsnot attempted1 (the original, redelivered after lock expiry)
okfails (lock expired)2: the copy and the redelivered original

The third row is the bug. The lock had expired, so complete was rejected. The copy was already scheduled, and the broker redelivered the original because nobody settled it. Now two independent instances of the same order exist.

It gets worse, because a scheduled copy is a brand-new message. The broker’s delivery count restarts at 0 on every copy, so you can’t use it to cap retries. I tracked attempts in a database column keyed by order. Both instances read and incremented that same counter:

  • The order’s three attempts were now shared by two instances, so it burned them twice as fast.
  • Read-then-increment wasn’t atomic. Two instances processing at the same moment both read 2, both failed, both scheduled another retry, and pushed the counter to 4. Both stayed alive.
  • Every retry spawned another copy per instance, so the duplication never collapsed.

What it looked like in production

Five reschedules failed with lock supplied is invalid within a four-second window. Those were exactly the five orders that were later dead-lettered, and each had two copies in the DLQ. About three minutes later, with the downstream still down, both copies of each order ran:

  • Three orders: one copy read counter 2 and retried, the other read 3 and dead-lettered.
  • Two orders: both copies retried and pushed the counter to 4.

A sequence view of one order:

original  → downstream call fails           counter 0 → 1
original  → schedule copy A (live now)
original  → complete FAILS (lock expired)
original  → redelivered 2 s later, fails     counter 1 → 2
original  → schedule copy B, complete OK
A and B   → both run ~3 min later, both read 2, both fail   counter → 4
A and B   → each schedules a 20-minute retry
...       → both reach the front of the queue, counter ≥ 3, dead-lettered

The fix: make it one operation

Service Bus supports transactions. A transaction can include a send (including a scheduled send) and a complete of the message you received. The broker applies them together or not at all.

private Mono<Void> scheduleCopyAndComplete(
    ServiceBusReceivedMessage original, ServiceBusMessage copy, OffsetDateTime when,
    ServiceBusReceiverAsyncClient receiver, ServiceBusSenderAsyncClient sender) {
  return Mono.usingWhen(
      sender.createTransaction(),                                       // begin
      tx -> sender.scheduleMessage(copy, when, tx)                      // op 1
          .then(Mono.defer(() -> receiver.complete(original,
              new CompleteOptions().setTransactionContext(tx)))),       // op 2
      sender::commitTransaction,                                        // both ok
      (tx, err) -> sender.rollbackTransaction(tx),                      // either failed
      sender::rollbackTransaction);                                     // cancelled
}

Mono.usingWhen ties the transaction’s fate to the outcome of the work inside it: commit on success, roll back on error or cancellation.

ScheduleCompleteCommitResultLive instances
okokokCopy visible at the scheduled time; original removed1
okfails (lock expired)rolled backCopy discarded; original redelivered1
failsnot attemptedrolled backOriginal redelivered1
okokfails or times outBroker aborts; original redelivered1

Every row ends at exactly one live instance. That’s the property I actually wanted.

The constraints nobody mentions

I hit most of these the hard way.

  • A transaction is bound to one AMQP connection. In the Java SDK, clients share a connection only if they’re built from the same ServiceBusClientBuilder instance. My sender and receiver each created their own builder, so they couldn’t participate in one transaction. Fix: one shared builder, and rebuild the receiver from that same builder after a failure.
  • Not available on the Basic tier. Standard and Premium only. On Basic, createTransaction fails, which would have made every retry fail. Check every environment before you ship.
  • One entity, unless you enable cross-entity transactions. Receiving from and scheduling to the same queue stays within one entity, so this was fine. A dead-letter replay that spans the DLQ and the main queue is a different story.
  • Transactions time out after two minutes. Mine wraps two quick operations. The slow downstream call happens before the transaction begins. Don’t put your slow work inside it.
  • Cost: two extra round trips (begin and commit), paid only on the retry path. Negligible next to a 16-second downstream call.

Checking it against a real broker

Unit tests proved the wiring: same transaction context on both operations, commit on success, rollback on a failed complete or failed schedule. But a mock can’t prove what the broker does, so I ran it against a real namespace with the downstream simulated. I held each message 400 seconds after delivery so every lock expired before the call, with the downstream returning connection resets throughout.

TimeWithout the fixWith the fix
6 min 40 sAttempt 1 fails; copy live immediately; complete fails. 2 liveAttempt 1 fails; broker rejects the transaction with amqp:transaction:rollback, copy discarded. 1 live
11 min 52 sAttempt 2 by redelivered original. 3 liveAttempt 2, rolled back again. 1 live
13 min 21 sAttempt 3 by first copy. 4 live
End of watchRetries exhausted, 4 live2 downstream calls, 1 live, 0 scheduled

Without the transaction, all three attempts were spent in under seven minutes. With it, the broker’s rollback discarded every copy.

What this does not fix

I want to be precise here, because this is where blog posts usually overclaim.

  1. A lost lock still costs an attempt. The redelivered original is processed and counts against the retry budget. The transaction prevents the second instance, not the redelivery.
  2. The success path is untouched. During the same test, 125 complete calls failed with an expired lock after a successful downstream call. Service Bus redelivered those, and at least 116 orders were submitted twice. The downstream system answered DUPLICATE_ORDER, which my code records as success, so no order was lost, but the downstream did duplicate work. A transaction doesn’t help: there’s no second operation to make atomic. The real fix is preventing the lock loss (the next post) and idempotency downstream.
  3. The counter is still read-then-increment. With a single live instance it stops mattering in practice, but it should be one atomic database update.
  4. Validated small. The real-broker runs used two to ten orders on one pod. I haven’t yet re-run the full outage under load.

Lessons

  • “Do A, then B” is a distributed-systems smell wherever A is visible to other consumers before B completes. If B can fail, A needs to be undoable.
  • Peek-lock settlement can fail on a message you handled perfectly. Treat complete as fallible, not as a formality.
  • Check for the same pattern elsewhere. In the same codebase, other listeners did scheduleMessage(...).subscribe() fire-and-forget followed by a complete. That’s worse: the two run concurrently, so a failed schedule plus a successful complete loses the message with only an error log to show for it. Chain the operations at minimum; use a transaction where you can.
  • Make the downstream idempotent. Everything I did reduces duplicates. Only idempotency makes them harmless.