Connecting two systems is not difficult. Both have APIs, the fields map, the demo works. Integrations are not judged on the demo.

They are judged eighteen months later, when finance asks why three orders in March never reached the ERP, and the answer takes a day to find because nothing logged the attempt.

The five failures to design for

Every one of these will happen. The only question is whether the system handles it or loses the record.

  1. The other system is down. Not permanently, just for ninety seconds during a deploy. Without retries the record is gone.
  2. The record is malformed. A field the sender considers optional, a character set nobody expected, a number formatted with a comma. One bad record must not stop the queue.
  3. The job runs twice. Retries, manual re-runs, two workers picking up the same message. Without idempotency you get duplicate orders, and duplicate orders reach customers.
  4. The network times out after the write but before the acknowledgement. The receiving system has the record; the sending system believes it failed and will send again. This is the case that produces the strangest bugs.
  5. The systems drift. Nothing failed loudly, but after six months the two sides disagree on a few hundred records. Without reconciliation you find out from a customer.

What to specify, concretely

These belong in the specification, not in a hardening phase after launch, because retrofitting them means changing the data model.

  • Idempotency keys on every write, so a repeated request is recognised rather than re-executed.
  • Retry with exponential backoff and a cap, so a brief outage self-heals and a long one does not hammer a system that is already struggling.
  • A dead-letter destination for records that fail repeatedly, plus an alert. Silent failure is the actual enemy.
  • Structured logging of every attempt – identifiers, timestamp, outcome, and the reason on failure. Log the operation, not the payload; payloads contain personal data.
  • Scheduled reconciliation comparing both sides and reporting divergence.
  • Alerting with an owner. A dashboard nobody has open is not monitoring.

Reconciliation is the part people skip

Retries and logging get built because they are visibly part of the work. Reconciliation gets cut because it delivers nothing on launch day.

It is the thing that tells you the integration is still correct. A nightly job that counts records on both sides, compares a checksum of the fields that matter, and emails a divergence report takes a day to build. Without it, the first evidence of a problem is a customer complaint, and by then the drift is months old.

It also changes the conversation when something does go wrong: instead of an argument about whose system is at fault, there is a report showing when the two sides stopped agreeing.

When not to build one

Not every connection deserves a custom integration. Off-the-shelf integration platforms are genuinely good for standard connectors and simple field mappings, and they come with retry and logging out of the box.

They become expensive when the logic is specific to your business, when the volume makes per-operation pricing painful, or when you need behaviour their model does not express. That is a cost and complexity judgement, not an ideological one.

And before either: measure the manual effort you are replacing. Automating a process that runs twice a month, takes ten minutes, and changes every quarter is usually the wrong investment. The right answer there is a better spreadsheet.