Trex Pi
Somewhere in every integration contract is a sentence like "each order will be transmitted exactly once." It has never once been true. The network partitioned mid-request and the sender retried; the queue redelivered after a consumer crash; a warehouse operator pressed the export button twice. Duplicates are not a failure mode - they are weather.
Idempotency keys, chosen carefully
The fix is old and unglamorous: every message carries a key, and processing is a no-op when the key has been seen. The interesting decisions are all in choosing the key.
- Natural keys beat generated ones. An order sync keyed on
order_id + statussurvives the sender regenerating its outbox; a random UUID minted at send time does not. - Scope the key to the effect, not the message. If one message creates an invoice and books stock, those are two effects with two keys - otherwise a partial failure leaves you unable to safely retry either.
- Expire keys deliberately. A dedup store that grows forever is a time bomb; one that expires too soon reintroduces duplicates. We default to 3× the sender's maximum retry horizon.
The test that matters
Every integration we ship passes what we internally call the double-tap test: replay yesterday's entire message log against today's system, twice, and diff the resulting state. If anything changed on the second pass, the pipeline is not done.
It is a brutal test and it fails constantly during development - on timestamps, on "created by" audit fields, on sequence counters. Every failure is a place where a 3 a.m. redelivery would have quietly corrupted data. Far cheaper to catch it in development than in production.