When Retries Create Duplicate Business Events
Retries are a sensible response to unreliable connections. If an API times out or a message broker cannot reach its consumer, sending the same request again may be the quickest way to restore flow.
The difficulty is that a failed response does not prove that the business action failed. In production, a retry can turn one intended action into two material postings, two status updates, or two requests for the same work.
1. A Timeout Describes Communication, Not Outcome
A timeout tells the sending system that no answer arrived within the expected period. It says nothing definitive about what the receiving system did before the connection closed.
The receiver may have committed a material issue, created a work order, or changed an approval status and then failed while returning the response. Repeating the call treats uncertainty as failure. That assumption is convenient for transport logic but unsafe for business state.
2. Delivery and Execution Are Different Guarantees
Integration platforms often describe delivery guarantees such as at least once or at most once. These terms matter, but they do not define how the application handles a repeated business instruction.
At-least-once delivery accepts that a message may arrive more than once. The consumer must therefore recognise whether the business event has already been applied. Reliable delivery without that recognition can produce reliably duplicated transactions.
3. Duplicate Payloads Are Not Always Duplicates
Comparing payload content appears to offer a simple defence. If two messages contain the same asset, quantity, timestamp, and status, the second can be rejected. That works only when identical content always represents one event.
Two technicians may legitimately book the same labour quantity against the same order. A repeated inspection reading may also be valid. Conversely, a retry may contain a new transmission timestamp and still represent the original event. Equality of fields is not equality of business intent.
4. Identity Must Survive the Retry
The sender needs a stable event or command identifier that is created before the first attempt and reused for every retry. A newly generated message ID on each attempt identifies transmissions, not the business action behind them.
The receiver can record that stable identifier with the result: material document number, created request ID, accepted work-order version, or explicit rejection. When the same identifier arrives again, it can return the recorded outcome instead of executing the action twice.
5. Database Constraints Help but Do Not Define Meaning
A unique constraint is useful when the stable business key can be enforced in one database transaction. It closes race conditions that an application-level lookup may leave open when two attempts arrive together.
But the key still needs a business definition. A uniqueness rule built from convenient fields may suppress valid events or allow duplicates whose formatting differs. The database can enforce a decision about identity; it cannot decide what one business event means.
6. Retries Need Boundaries and a Recovery State
Unlimited retries do not create resilience. They increase load, obscure the original failure, and may repeat an unsafe action long after its operational context has changed. A status update replayed after a work order was manually reopened can overwrite a legitimate later decision.
Retries should be bounded by count, time, and eligible error type. Once that boundary is reached, the transaction needs a visible unresolved state. An owner must be able to inspect the last known outcome, compare both systems, and choose replay, correction, or cancellation.
7. Reconciliation Finds What Idempotency Cannot Prevent
Idempotency protects repeated processing when every participant preserves and respects the event identity. Production landscapes rarely meet that condition everywhere. Legacy imports, manual database corrections, batch aggregation, and downstream fan-out can break the chain.
Reconciliation remains necessary to compare business results across boundaries. It should find, for example, one outbound material issue linked to two receiving documents, or an acknowledged work-order update with no matching target status. Retry control prevents a class of duplicates; reconciliation exposes the remaining divergence.
8. A Practical Minimum
For every interface that creates or changes business records, document the unit of business intent and assign it a stable identifier before transmission. Preserve that identifier across retries, log each attempt separately, and store the receiving outcome against the stable key.
Define which errors are retryable, the maximum retry window, and the state used when the outcome is uncertain. Test a lost response after a successful commit, two simultaneous deliveries, and a retry after a later manual change. Name the role authorised to resolve each unresolved transaction.
Conclusion
Retries are useful because temporary communication failures are normal. They become dangerous when the integration assumes that no response means no business action occurred.
Technical retry repeats a transmission. Business idempotency preserves the meaning of one intended action across repeated transmissions. A reliable interface needs both, plus a recovery path for the cases where the outcome remains uncertain.
Comments
Post a Comment