Perspective / Software & CRM
Reliable CRM integration: designing for duplicate events, partial failure and replay

Reliable CRM integration requires a clear account of what happens when only part of a business process succeeds. A customer update may be committed while its event is not delivered; a consumer may finish its work before acknowledging a message; a retry may repeat an external action. The architecture must handle these states deliberately. This article develops an illustrative event-based order flow, explaining where atomicity ends, how duplication is controlled and what evidence is needed to operate and recover the process.
The executive view
- Choose synchronous checks and asynchronous events according to business commitments and delay tolerance.
- An outbox does not remove duplicate delivery or external transaction boundaries.
- Test replay, ordering and recovery with business-level evidence and named operational ownership.
Choose synchronous APIs and events from the business requirement
Use a synchronous request when the caller needs an authoritative answer before making a commitment. Use an event when a completed change should inform independently operating consumers. Some journeys need both: an immediate eligibility check followed by asynchronous updates after the order is accepted.
Do not introduce an event broker merely to make an architecture appear modern. Asynchronous processing adds pending states, monitoring, recovery and support responsibilities. Those costs may be justified by independent processing, variable demand or multiple consumers, but they are real operating obligations.
Specify the commercial tolerance for delay. The CRM’s sales history may accept a short lag; a commitment involving credit or scarce stock may require a fresh check. Record the permitted information age, the decision that depends on it and the behaviour when the limit is exceeded. This connects integration design to an explicit business risk rather than an unexplained preference for a technology.
An integration can combine both models. Validate credit synchronously before accepting an order, then publish the accepted outcome for reporting and customer-service views. If the credit service is unavailable, decide whether the user receives a pending application, a refusal or a bounded exception; do not let an accidental timeout policy make that commercial choice. State the maximum acceptable delay for every asynchronous consumer. A CRM view used during a sales call may tolerate seconds of staleness, while a financial close process may tolerate longer provided completeness is demonstrable. Consistency requirements should follow those distinct commitments.
Define events as contracts with stable meaning
An event should describe a business fact that has occurred, such as an order being accepted. A command asks another component to perform work. Keeping the distinction clear helps prevent an apparently informational message from becoming an undocumented remote instruction.
Define an envelope containing an event identifier, entity identifier, schema version, relevant entity version, occurrence time and correlation identifier. Explain the business meaning of the payload and whether it represents a historical fact or a pointer to current state. A consumer retrieving current information may receive values that differ from those present when the event occurred.
Document compatibility rules before adding consumers. Changes to vocabulary, optional fields or state transitions can have wider consequences than a simple JSON-schema change suggests. Establish how consumers are tested, how old contracts are retired and how the business will identify incompatible processing during a rollout.
Keep occurrence time, publication time and receipt time distinct. An order accepted yesterday but delivered today should retain yesterday's business timestamp. Include the producing system, entity reference, event identity and schema version, with a documented policy for optional and unknown fields. Events that describe a fact should avoid relying on a consumer immediately fetching a mutable current record to reconstruct that historical fact. Conversely, copying an entire customer record into every event creates unnecessary disclosure and versioning obligations. Publish the minimum stable information needed for the consumer's declared decision.
Close the database-and-publish gap
A common failure appears when an application updates its database and separately publishes an event. If one operation succeeds and the other fails, the application and downstream systems disagree. The transactional outbox pattern addresses that local dual-write boundary by recording the business change and an outgoing event within one database transaction.
A relay then publishes committed outbox records. That relay can still publish a record more than once, particularly if it fails after publishing but before recording completion. The outbox therefore supports reliable handover; it does not remove the need for duplicate handling.
Where change data capture is appropriate, assess its capture semantics, retention, ordering and operational ownership. The design choice depends on the database, the required event meaning and the surrounding process. Raw database changes and deliberately authored business events are not automatically equivalent contracts.
Operational details determine whether an outbox remains reliable under load. A relay needs a bounded batch size, a safe way to claim work across competing workers and a recovery policy for abandoned claims. Published records need retention long enough for diagnosis and agreed replay, followed by deliberate cleanup. Track the oldest unpublished record as well as queue depth: a small stuck backlog can affect important customers. Monitor storage growth and producer transaction latency so a failing relay does not eventually turn an integration incident into an outage of the originating business application.
Worked example: the relay publishes the same order twice
In this illustrative flow, an order service commits order O-1042 and outbox event E-77 together. The relay publishes E-77, then crashes before marking it as published. On restart, it publishes the same event again.
| Component | Required behaviour |
|---|---|
| Order database | Preserve one accepted order and the committed outgoing event |
| Relay | Retry delivery with the original event identity |
| Finance consumer | Recognise an already processed event or business operation |
| CRM consumer | Avoid creating duplicate order-history entries |
| Operations tooling | Show delivery attempts and the eventual processing outcome |
For a consumer writing to its own database, a unique processed-event record can be committed alongside the local effect. The duplicate then becomes a recognised no-op rather than another invoice or history record.
A separate business-operation key may also be necessary: two distinct messages can refer to the same intended commercial action. The example demonstrates a design expectation, not a claim that duplication can be ignored at the broker.
In a hypothetical hour, the producer accepts 10,000 orders and the relay makes 10,120 delivery attempts because of retries. That is not evidence of 120 additional orders or even 120 duplicates processed. Reconcile distinct event IDs, acknowledged deliveries and committed consumer effects separately. If the finance service has 9,995 effects and five recognised pending events, the current discrepancy has an explanation. If it has 9,995 effects and no remaining events, successful transport metrics conceal missing business work. The acceptance test should demonstrate these distinctions through queryable evidence, rather than relying on a reassuring total-message count.
Treat external actions as separate transaction boundaries
A local database transaction cannot atomically include an arbitrary external payment or fulfilment API. If the consumer makes the external call and then loses its connection, it may not know whether the action completed. Retrying blindly can create a duplicate commitment.
Use a stable operation identity and the external service’s documented idempotency mechanism where available. Check its scope, retention and response semantics. Stripe’s API documentation provides a concrete example, but the behaviour of one provider should not be assumed for another.
Where an action cannot safely be repeated, provide a status-query or reconciliation route and a visible pending state. A timeout should not automatically be interpreted as failure. Capture enough evidence to determine the outcome without exposing unnecessary customer data in logs. The runbook needs an owner authorised to resolve uncertainty about the business commitment.
Check the duration of the provider's idempotency guarantee against the longest plausible recovery period. If a retry occurs after that window, the same local key may no longer protect the external action. Persist the external operation reference and query it where supported before repeating an uncertain request. Distinguish a rejected request from an accepted request whose response was lost. Reconciliation may need provider settlement records or fulfilment evidence, rather than just the last HTTP status. These checks are especially important when an operator is asked to clear a backlog after a multi-day incident.
Design replay, ordering and compensation before an incident
A replay can repair an incomplete read model or apply corrected processing, but it can also repeat side effects. Distinguish rebuilding a projection from re-executing a commercial action. Define which consumers may replay, which effects are suppressed and how historical contracts remain interpretable.
Ordering requirements should be explicit for the relevant entity or process. A customer suspension arriving after a later reactivation must not accidentally restore the older state. Version checks, state-transition validation and authoritative lookups can help, depending on the contract.
For multistep processes, decide whether failure requires retry, compensation or human review. Compensation is a new business action, not necessarily a perfect reversal. Returning a payment, cancelling a shipment and restoring an earlier record each have different constraints. A dead-letter queue is useful evidence, but placing a message there does not resolve the affected customer or financial outcome.
Partition ordering by the narrowest entity that requires it. Global sequencing can create a bottleneck without protecting any additional business invariant. For an individual account, attach an expected revision and specify how the consumer handles a gap, a stale event or a conflicting transition. A replay tool should offer a dry-run impact view, preserve original identifiers and record who initiated it. Suppress emails, charges and external orders when rebuilding a read model unless those effects are explicitly authorised. Historical payloads may require maintained converters; deleting an old handler can turn archived events into unusable evidence.
Include deletion and correction semantics in the event model. A customer record disappearing from a source system may mean an authorised deletion, a merge or an extraction defect; consumers should not infer the same action from all three. Specify which records must be removed, which financial or operational facts remain under the applicable retention policy and how derived views are corrected. A replay must respect those later decisions rather than resurrecting an obsolete profile from historical events. Restrict access to retained payloads and document how recovery tooling handles superseded sensitive information.
Specify the integration acceptance contract before procurement
An integration tender should describe observable business behaviour, rather than only requiring webhooks or a message broker. State the source of truth, eligible event population, maximum normal delay, permitted inconsistency and recovery objective. Ask bidders to show how they will handle duplicates, missing deliveries, out-of-order changes and uncertain external outcomes. These requirements make competing technical approaches comparable.
Use a controlled test ledger containing expected source events and downstream effects. Inject a relay crash, a delayed consumer, an invalid payload and a provider timeout after acceptance. Require an explanation of every resulting discrepancy and a demonstrated recovery path. The handover should include access to the evidence, documented replay permissions, alert ownership and dependency limits. A demonstration in which every component remains healthy cannot establish these properties.
Evaluate total operating cost alongside build cost: retained event storage, broker charges, reconciliation, support cover and contract-version maintenance all continue after launch. A simpler scheduled integration can be the better commercial choice when its delay and recovery characteristics meet the need. Event-driven architecture is justified when its decoupling and responsiveness are worth the additional responsibilities. The supplier's proposal should make that trade-off visible instead of presenting asynchronous messaging as an automatic maturity upgrade.
Operate the process through business-level evidence
Monitor more than broker availability. Useful measures include the age of unprocessed business events, reconciliation differences, repeated failures, time spent in pending states and the number of cases awaiting an authorised decision. Correlation identifiers should connect evidence across producer, transport and consumers.
Test the architecture by injecting failures at meaningful boundaries: after database commit, after publication, after an external action and before acknowledgement. Demonstrate that the system reaches a defined state and that operators can identify what happened. Keep duplicate, out-of-order and replay cases in the acceptance evidence.
Microsoft’s event-driven architecture guidance discusses the associated consistency and operational trade-offs. The executive decision is whether those obligations fit the business’s capability and exposure. A reliable integration is an operated process with defined recovery, not merely a diagram of connected applications.
Establish recovery capacity as well as normal throughput. Suppose an outage accumulates 60,000 events while new work continues at 2,000 events per hour. A consumer processing 5,000 per hour has 3,000 per hour of spare capacity, so the nominal drain time is twenty hours, before allowing for failures and rate limits. That calculation can invalidate an otherwise attractive recovery promise. Test realistic downstream limits and prioritisation rules. A backlog of customer-critical updates may need a different queue from analytics refreshes. Make those choices in advance so incident responders do not invent business priorities under pressure.
Questions this raises
Does an outbox guarantee exactly-once business processing?
No. It closes a local database-and-publish gap, but delivery can still repeat. Consumers need suitable duplicate handling, and external side effects require their own idempotency, status and reconciliation arrangements.
When should CRM integration use synchronous APIs?
When the caller needs an authoritative answer before making a commitment and the availability trade-off is acceptable. Events are useful for informing independent consumers of completed changes, but introduce delay and additional operating responsibilities.
Is a dead-letter queue a complete recovery strategy?
No. It isolates messages that could not be processed. The business still needs to understand the affected operation, decide whether to retry or compensate, and verify that customer and financial records reach a coherent state.
Sources & further reading
From thinking to doing