Your database saved the order. Did the event get published?

The order exists, but the next step never starts

Imagine a checkout service that writes an accepted order into its database and then publishes an OrderAccepted event. A warehouse service listens for that event to begin fulfillment. During a short broker outage, the database write succeeds and the publish call fails. The customer sees an order number, but the warehouse never receives the instruction that makes the order useful.

Reversing the calls creates a different failure. If the checkout service publishes first and its database transaction later rolls back, the warehouse may receive an event for an order that does not exist. Adding a retry can help with temporary failure, but it does not make two independent systems commit one decision together.

The transactional outbox pattern changes where the service records its obligation to publish. The business change and an outbox record are committed in one database transaction. A separate relay reads committed outbox entries and sends messages to the broker. The database is the local source of truth for both the accepted order and the work that must leave the service.

This guide uses a fictional orders system to explore the design. All quantities, incident scenarios, and acceptance tests are illustrative. AWS documents the pattern and its delivery considerations. The schemas, operating questions, and rollout exercises here are editorial design examples that require adaptation to your database, broker, and business rules.

First decide whether an event is the right contract

Before building an outbox, ask what the receiving service actually needs. An event describes something that happened. A command asks a recipient to do something. Those distinctions affect naming, ownership, and how failures should be handled. OrderAccepted is a statement owned by checkout. ReserveInventory is a request with a different responsibility and response model.

A warehouse consumer may need only enough information to start its workflow. It does not necessarily need a copy of every order field. Define which attributes are part of the public event contract, which are private database details, and which can be retrieved through an authorized interface if needed. Smaller, intentional contracts are easier to evolve and protect.

Consider whether the business can tolerate asynchronous progress. The checkout service may accept an order while inventory confirmation happens later, but the customer experience must reflect that state honestly. If acceptance legally or commercially requires a reservation first, the workflow needs a different decision boundary. An outbox cannot repair a business contract that promises more than the system has established.

Write the event's meaning in one sentence and give it an owner. List the consumers that rely on it and the behavior expected when delivery is delayed. This small contract review prevents a reliable relay from distributing ambiguous messages more efficiently.

Put the business change and publication intent together

Within the local transaction, write the accepted order and an outbox row describing the event. Commit both or neither. If the transaction rolls back, the relay should have no committed publication intent to process. If it commits, the service can recover the pending event even if the application process exits immediately afterward.

Keep the transaction boundary narrow. It should contain the database operations required to establish the business fact and its publication record. Calling the broker while holding the transaction open reintroduces an external dependency into that boundary and can lengthen locks or contention. The relay exists to decouple the external send from the local commit.

Confirm that your database arrangement really supports the atomicity you expect. Two writes in different databases or unsupported transaction scopes do not become atomic because application code places them in one function. Review the actual storage engine and transaction model, and test rollback with the same tables and connection behavior used by the application.

For the hypothetical order, the acceptance test is direct. Force an error after the order write but before commit. Neither the accepted order nor its outbox entry should remain committed. Then commit successfully and terminate the application before it sends anything externally. The outbox entry should remain available for later delivery.

BEGIN;

INSERT INTO orders (order_id, status)
VALUES ('demo-order-42', 'accepted');

INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES (
  'demo-event-42',
  'demo-order-42',
  'OrderAccepted.v1',
  '{"order_id":"demo-order-42"}'
);

COMMIT;

Design the smallest useful outbox record

A practical outbox entry needs a stable event identifier, event type, contract version, aggregate identifier, payload, and creation time. Operational fields depend on the relay design. A polling relay might also track attempt state or a lease. A change-data-capture approach may use database log positions instead of a mutable sent flag.

The event identifier is particularly important. Generate it as part of the original business decision and retain it across publication retries. A relay that creates a new identifier every time it sends the same event makes duplicate handling harder for consumers. Distinguish an event's identity from the identifier of an individual delivery attempt.

Use a contract version to explain payload shape or semantics, but do not assume that adding a number makes all changes safe. A consumer still needs rules for fields it does not recognize, required fields that are absent, and event types it cannot process. Maintain examples that represent both the current and supported older contracts.

Avoid treating the outbox as a copy of the entire source row by default. Include data deliberately. Order notes, payment details, or personal contact information may not belong in a broadly distributed event. The outbox is part of your data distribution surface, so apply the same ownership and retention reasoning used for other exported business records.

Follow one event through a polling relay

A polling relay periodically finds pending committed entries, claims a bounded set, sends them, and records the outcome. The exact claim mechanism must be supported by the database and safe under concurrent workers. A simple query that reads all pending rows without coordination can let several workers publish the same backlog at once.

Keep the batch size bounded and observe how long a batch takes. Large batches can monopolize a worker and delay recently created events. Very small batches can add unnecessary database overhead. Choose a starting point from measured behavior and maintain a clear relationship between polling frequency, database load, and acceptable publication delay.

Do not hold database locks across a slow broker request unless you have deliberately accepted that trade-off and tested its consequences. A lease or claim record can separate ownership from the external send, but it introduces its own expiry and recovery rules. The operator must be able to distinguish an active worker from abandoned work.

For the orders service, record the oldest pending event and the number of events awaiting publication. Those two measures answer different questions. A small backlog can contain one critically old order, while a large recent burst may clear within the agreed delay. Build alerts around the business impact of age and progress.

Understand the uncertain publish outcome

A publish timeout does not prove that the broker rejected a message. The broker may have accepted it while the acknowledgment was lost. If the relay marks the entry sent without confirmation, an actually rejected event may disappear. If it retries, an accepted event may be delivered again.

This is why the design should expect duplicates. The outbox protects the local obligation to publish, but the send and its local completion record still span different systems. AWS explicitly notes duplicate handling as an implementation consideration. The receiving workflow needs an intentional response when the same event arrives more than once.

Consider a concrete sequence. The relay sends event E42, the broker accepts it, and the relay crashes before updating local state. After restart, E42 still looks pending. The relay sends E42 again using the same identifier. A consumer that tracks E42 can recognize the replay and avoid repeating a completed local business effect.

Make this failure a routine test rather than a surprising production discovery. Stop the relay at the acknowledgment boundary in a sandbox and inspect what the broker, outbox, and consumer each believe happened. That exercise reveals whether the system is designed around delivery uncertainty or merely hopes the happy path will be common.

Make the consumer's local effect repeatable

A consumer that updates its own database can often record the processed event identifier in the same transaction as its business change. For the warehouse, that might mean creating a fulfillment task and recording E42 as processed together. A unique constraint on the recorded identity can help prevent two concurrent handlers from both treating the event as new.

The transaction boundary matters here too. Recording E42 before creating the task can lose work if the process fails between the operations. Creating the task before recording E42 can duplicate work on retry. Place the local decision and its duplicate marker within the supported atomic scope, and define how the handler responds to an already processed event.

Keep the deduplication key aligned with the effect. Some events legitimately trigger several independent consumers. A global marker that suppresses all processing after one subscriber finishes would be wrong. The identity should distinguish the event and the particular consumer responsibility where that distinction is needed.

Decide how long duplicate records must remain useful. Retention should account for ordinary retries, dead-letter redrives, disaster recovery, and authorized historical replay. Deleting markers sooner than the replay horizon can turn an old, apparently harmless message into a new business action.

Handle side effects outside the consumer database

A database transaction cannot atomically include every external side effect. Sending an email, charging a payment provider, or calling a shipping API introduces another boundary. Marking an event processed locally does not prove that the external effect happened, and an uncertain external response can require careful reconciliation.

Use the external provider's supported idempotency mechanism when available, with a stable key tied to the intended action. Read its scope and retention rules rather than assuming a request key works forever. A provider may distinguish accounts, endpoints, or request bodies in ways that affect retries.

When no suitable idempotency mechanism exists, design an explicit action record and reconciliation path. The handler can record intent, attempt the external operation, and retain enough evidence for an operator to investigate uncertainty. That may be more work than a simple retry loop, but it reflects the business risk of repeating the action.

For the warehouse example, printing a shipping label twice may incur cost even if duplicate database tasks are prevented. List such effects before declaring the consumer idempotent. The useful question is whether each business consequence can be safely repeated, detected as already complete, or reconciled with a documented owner.

Choose the ordering boundary that the business needs

Global ordering is usually a stronger requirement than a business process needs. The warehouse may need changes for one order in sequence while processing different orders independently. Define the aggregate whose state transitions must be interpreted together, then map that boundary to the broker and relay features you choose.

Give events a meaningful version or sequence within that aggregate if the consumer needs to detect gaps or stale updates. A timestamp alone can be ambiguous when several updates occur close together or clocks differ. The source of ordering should match how the authoritative business state is committed.

Decide what a consumer does with an older event after a newer one has already been applied. It might ignore an obsolete status, retain it for audit, or stop and reconcile a missing transition. That is a contract decision. The relay cannot infer whether OrderCancelled supersedes a fulfillment action that has already left the building.

Test concurrent updates to the same order. A design that preserves sequence in a single-threaded demonstration may reorder work when multiple relay workers run. Verify the complete path through claiming, publishing, broker partitioning, and consumer concurrency rather than attributing the guarantee to one component.

Compare polling with change data capture

Polling can be a straightforward first implementation when the expected volume and delay fit the database workload. It makes pending publication visible in application-owned tables and can be operated with familiar queries. Its costs include repeated reads, claim coordination, and cleanup of processed records.

Change data capture can move committed changes through a database log or supported stream. That can reduce polling work and integrate with an existing data platform, but it adds dependencies on connector behavior, offsets, retention, and recovery. Review the supported transaction and ordering semantics for the actual connector.

The decision should reflect the team's operating capacity. A sophisticated streaming pipeline that nobody can restore may be riskier than a bounded polling relay with clear ownership. Conversely, a heavily loaded database may be a poor place for an aggressive polling pattern that ignores existing infrastructure.

Write down how each option recovers from an outage longer than its normal operating window. What position identifies progress? What happens when source log retention is exceeded? Can the team reconstruct which events need delivery? Those questions are often more decisive than a benchmark of steady-state throughput.

Keep retry policy separate from business urgency

A retry policy should distinguish a temporary delivery problem from a malformed or unauthorized event. Repeatedly sending an invalid payload at high frequency consumes resources without increasing the chance of success. A broker outage needs a different response from a contract validation failure.

Use bounded backoff and observable attempt records appropriate to your relay. Preserve the stable event identifier and the original creation time. The age of the business obligation should not reset every time the relay makes another attempt, or an old order can look perpetually fresh in monitoring.

Define an escalation threshold in business terms. Perhaps the team must investigate an accepted order whose publication delay threatens the promised fulfillment window. That threshold is an editorial example, not a universal duration. Different workflows have different tolerance for lag, and some need human review before a delayed action is resumed.

Keep a controlled path to resume quarantined work after the underlying problem is fixed. An operator should know which change makes the event safe to retry and which evidence proves it was not already completed. A retry button without context can create the very duplicates the outbox was designed to manage.

Give poison messages a visible destination

A poison message repeatedly fails because its content or processing conditions are incompatible with the consumer. It may refer to an unsupported contract version, violate a business invariant, or expose a bug in the handler. Continuing to retry indefinitely can block progress for unrelated work.

Create a quarantine or dead-letter process that preserves the event identity, reason for failure, relevant timestamps, and authorized inspection path. Avoid copying sensitive payloads into a broadly accessible ticket. The operational record should help the responsible team diagnose the issue without expanding data exposure unnecessarily.

For an ordered aggregate, decide whether later events for the same order may proceed while one event is quarantined. Skipping the failed event can break state assumptions. Blocking every order because one aggregate failed can create an unnecessarily large incident. The right boundary belongs in the consumer contract and operating procedure.

Rehearse a redrive. Fix a sandbox handler, reprocess the quarantined event, and confirm that previously completed effects remain deduplicated. Record which operator action resumes the flow. Recovery is part of the delivery design, not an optional administrative feature to add after the first backlog.

Observe the age of business work

A relay can have excellent process uptime while failing to publish useful events. Measure the age of the oldest unpublished record, the rate of successful publication, the growth of pending work, and recurring failure categories. Connect those measures to the service that owns the business obligation.

Consumer monitoring needs a different view. Track whether events are being applied, ignored as duplicates, rejected for contract reasons, or waiting for reconciliation. A high duplicate rate can be expected during recovery, but a sudden change may reveal relay instability or an accidental replay. Interpret the signal in context.

Correlate the order identifier, event identifier, delivery attempt, and consumer action. Keep their roles distinct in logs. If every identifier is labeled request ID, an operator will struggle to follow the event across asynchronous boundaries. A small naming standard can save time during a real incident.

Build one dashboard around a concrete question: which accepted orders have not progressed to the next valid state within their expected window? Infrastructure metrics help explain the answer, but the business state reveals the impact. Use both views rather than expecting queue depth alone to represent customer experience.

Plan cleanup before the table becomes an archive

Processed outbox records accumulate. Decide whether they are operational retry state, an audit record, or a temporary publication buffer. Those purposes have different retention needs. Treating one table as all three without a plan can make cleanup politically difficult and technically risky.

Delete or archive in bounded batches consistent with your database's behavior. Test the effect on locks, indexes, replication, and ordinary transaction latency. A cleanup task that runs infrequently and removes a huge volume at once can create an incident even when the relay itself is efficient.

Preserve enough evidence to reconcile recent publication problems. If records disappear immediately after a successful send, an operator may have little context when a consumer reports a missing order. Retention should cover the realistic investigation window and the privacy obligations of the payload.

Document which history can be replayed after cleanup. If you retain only business state, reconstructing an old event may produce a different payload from the one originally published. A replay strategy should explain that difference rather than promise an exact historical stream from incomplete data.

Treat event contracts as an interface

An event is an interface used outside the producer's immediate transaction. Adding a database column does not automatically justify exposing it. Removing a field or changing its meaning can break a consumer that still processes older records. Maintain a contract review process that includes the teams using the event.

Provide representative payload examples and explicit semantics for missing fields. For an order amount, specify currency and whether the value includes adjustments. For status, explain which transitions are possible and whether consumers should treat unknown values as invalid or forward-compatible. Ambiguous semantics cause business errors even when delivery is reliable.

Roll out compatible changes in an order the consumers can tolerate. A new consumer may need to accept both old and new forms during a transition. A producer may need to keep a field until every supported consumer has moved. The exact plan depends on your contract and deployment independence.

Test replay of older events against the current handler. A contract that works only for messages created after the latest deployment can fail during disaster recovery or backlog processing. Keep a small, sanitized collection of historical examples as part of the team's validation process.

Design a safe replay procedure

Replay is powerful because it can restore missing downstream state. It is dangerous because it can also repeat externally visible actions. Before replaying, identify the purpose, target consumers, event range, and expected effect. Rebuilding a search index and resending customer notifications are different operations.

Use stable event identities where replay is meant to represent the original fact again. If a new business decision is being made, create a new event with explicit semantics rather than disguising it as a delivery retry. Operators need to distinguish historical recovery from new work.

Start with a small, reviewable range and inspect consumer outcomes. Record the initiating operator, reason, source range, and completion evidence. Apply rate limits that protect downstream services and leave normal traffic with an acceptable share of capacity. A recovery job should not overwhelm the system it is trying to repair.

End with reconciliation. Count expected aggregate transitions or rebuilt records, inspect failures, and explain any skipped events. A command that finishes successfully only proves the tool completed its own loop. The business recovery is complete when the intended downstream state matches the agreed source of truth.

Reconcile the database with the downstream workflow

Even a well-designed delivery system benefits from reconciliation. Compare accepted orders with downstream fulfillment state using an authorized, bounded process. The aim is to detect a missing or inconsistent business transition that component-level metrics may not reveal.

Choose the comparison keys and time window carefully. An order created moments ago may legitimately be waiting for publication. An order cancelled before fulfillment may correctly have no shipping task. Encode those distinctions in the reconciliation criteria rather than treating every absent row as a delivery failure.

For a suspected gap, inspect the original business transaction, outbox entry, relay history, broker evidence, and consumer record. Follow the stable identifiers through the path. Avoid manually inserting a downstream task until the team understands whether a delayed original event may still arrive and duplicate the repair.

Assign ownership to discrepancies. Checkout owns whether the order was accepted, the relay owner investigates publication, and the consumer owner investigates local application. A shared process can coordinate them, but every gap needs a named next action. Otherwise, reconciliation produces a growing report without resolving customer impact.

Test the failure matrix deliberately

A useful test suite covers the boundaries where the system can disagree about progress. Test a rollback before local commit, a process exit after commit, a broker rejection, a lost acknowledgment, and a consumer crash around its local transaction. Each scenario should have an expected final state.

Also test concurrency. Run two relay workers and two deliveries of the same event against the consumer. Verify that coordination and duplicate handling work under overlapping execution, not just sequential retries. Use harmless sandbox effects so the test does not create real shipments, payments, or notifications.

Test recovery after a backlog forms. Stop the relay, create a bounded set of orders, restart it, and observe publication age and database load. Then stop a consumer and repeat the exercise on the receiving side. Producer and consumer recovery can stress different resources.

Capture the result as a state comparison rather than a green log line. For each order, identify the committed order row, original event identity, expected consumer action, and any unresolved uncertainty. The exercise should demonstrate eventual business correctness within the chosen operating conditions.

Estimate capacity with a bounded workload example

Suppose a fictional service commits sixty orders per minute during a normal period and a relay can publish ninety events per minute under the tested conditions. A temporary outage creates a backlog while new orders continue arriving. The spare rate available for draining that backlog is the difference between publication capacity and incoming work, not the full publish rate.

If six hundred events are pending and conditions remain stable, thirty events per minute of spare capacity suggests about twenty minutes to drain them. This is an illustrative arithmetic model. Real performance can change with batch size, broker limits, database contention, retries, and uneven processing cost.

Use the example to ask a better question about recovery. Can the business tolerate the oldest event waiting through both the outage and the drain period? If not, the team may need more relay capacity, a different backlog strategy, or a revised service commitment. Ordinary steady-state throughput does not answer that question.

Repeat the measurement with realistic payloads and downstream behavior. Publishing a tiny synthetic event to an idle broker can overstate useful capacity. Include the checks, acknowledgments, and state updates the production relay performs, then document the conditions under which the estimate is valid.

Secure the event path and its operational tools

Restrict which applications can write outbox entries and which relay identity can publish them. A broad database role or broker permission can turn an integration path into an unintended data distribution mechanism. Align authorization with the producer's actual event types and destinations where supported.

Protect inspection and replay tools as carefully as the ordinary publisher. A convenient administrative endpoint that accepts arbitrary payloads or destinations can bypass the contract controls in the application. Require a clear purpose, bounded scope, and an audit record for operational actions.

Review the payload before enabling broad logging. An order event may contain personal information or commercially sensitive attributes. Prefer stable identifiers and sanitized failure details in ordinary logs, with restricted access to payload inspection when needed. Observability should make recovery possible without casually duplicating sensitive records.

For a tabletop, ask what happens if someone publishes an unauthorized OrderAccepted event. Can the consumer validate the expected source and required contract? What prevents an operator from replaying all historical orders into a live fulfillment destination? The answers should be concrete controls and reviewed procedures.

Introduce the pattern without creating two publishers

During migration, an old direct publish path and a new relay can accidentally emit the same business fact with different identifiers. Plan the transition so each committed decision has a clear publication owner. A feature flag is useful only if its behavior is understood across concurrent application versions.

Start in a sandbox with the original direct path disabled for the selected workflow. Verify that a committed order produces an outbox entry and that the relay publishes it. Then test a rolling deployment where old and new application instances can coexist. Decide which compatibility behavior is required during that interval.

Avoid silently writing an outbox row and continuing the old send as an untracked fallback. If both paths are intentionally active for a shadow test, use a non-production destination or explicit observation mechanism that cannot trigger duplicate business actions. Separate validation traffic from real fulfillment.

Define rollback in terms of publication state. If the relay is disabled after some orders committed through the new path, those pending entries still need an owner. Reverting the application does not erase their obligation. A migration runbook should explain how the team drains or safely retains that work.

Recognize when the outbox is the wrong amount of machinery

Not every application needs a custom relay and event contract. A small service with no asynchronous downstream effect may be better served by a simpler transaction and a direct response. An existing managed integration may already provide the guarantees the workflow needs, subject to its documented limits.

The outbox is valuable when a local business commit must reliably leave behind publication intent. It does not automatically coordinate independent service transactions, undo external actions, or decide how a multi-step business process compensates for failure. Those are additional design problems with their own contracts.

Ask whether the team can own the relay's backlog, cleanup, alerting, and replay procedures. A pattern selected only because it appears in an architecture diagram can create an unsupported service. Include operating effort in the decision alongside correctness and performance.

For the fictional checkout, the decisive requirement is that every accepted order has a recoverable path to downstream processing. If a simpler supported mechanism meets that requirement with clear failure semantics, evaluate it honestly. The goal is an operable business workflow with explainable behavior when components fail.

Review the design with the people who carry the consequences

Bring together the producer owner, consumer owner, database operator, and business representative. Walk through one order, one duplicate, one missing acknowledgment, one incompatible payload, and one replay request. Ask each person what they expect to see and what they would do next.

The business representative should be able to explain what a delayed order means for a customer. The consumer owner should identify which side effects require special protection. The database operator should understand polling and retention load. The producer owner should demonstrate that local rollback preserves the intended atomic boundary.

Record unresolved assumptions as specific tasks. For example, the team may need a stable event identifier, a consumer uniqueness constraint, a replay approval path, or a query for the oldest pending event. These are concrete implementation gaps, not reasons to declare the whole design vaguely unfinished.

Before launch, choose an initial observation window that includes a real release and a recovery rehearsal. Review actual publication age, duplicate handling, and consumer progress. Keep the evidence tied to the design decisions so future maintainers can understand which guarantees were tested.

A relay incident review in six questions

When publication age rises, first ask whether new outbox records are still being committed. If they are, the producer may be healthy while the relay or broker path is degraded. If they are not, inspect the business transaction path before focusing on the message system.

Second, ask whether workers are claiming records and whether those claims expire as intended. A worker can be alive while repeatedly observing work leased by a failed predecessor. Third, inspect the error categories for attempted sends. An authorization change, payload rejection, and temporary network failure suggest different next actions.

Fourth, determine whether the broker accepted any of the uncertain sends. Preserve event identifiers and attempt records for that investigation. Fifth, inspect consumer progress separately. A recovering publisher can fill a queue faster than a consumer can drain it, so publication success does not establish that the customer workflow has recovered.

Finally, decide how the team will verify the business backlog has cleared. Compare accepted orders with valid downstream state and inspect remaining exceptions. The incident ends when the agreed service condition is restored and unresolved work has an owner, not simply when the relay process restarts.

Write an operating agreement between teams

The producer and consumer teams should agree on event meaning, publication-delay expectations, supported contract versions, duplicate behavior, and replay ownership. Put those agreements close to the interface documentation so they change when the event contract changes.

Include an escalation path for a specific event identifier. A consumer owner should be able to ask whether E42 was committed and published without requesting a full database export. The producer owner should be able to distinguish a publication problem from a downstream business rejection using the shared evidence.

Agree on how breaking changes are announced and tested. An asynchronous boundary can hide the fact that a consumer is several releases behind until an old message is retried. A small compatibility window and representative historical examples make that situation easier to manage.

Review the agreement after the first real incident or recovery exercise. Use observed ambiguity to improve the next version. A dependable event path combines storage and broker behavior with clear ownership of the business work that crosses the boundary.

Preserve the distinction between delivery and completion

A broker acknowledgment establishes progress at one boundary. A consumer transaction establishes progress at another. A completed customer workflow may require several more actions. Keep those states separate in dashboards and incident language.

For an accepted order, the relay can be healthy while the warehouse is waiting on an external shipping provider. The right response follows the blocked business step. Stable event and order identifiers let teams coordinate that investigation without assuming that every delay is a missing message.

Start with one event and a complete recovery story

Choose a single business transition and define its event contract. Commit the business row and publication intent together, run a bounded relay, and make one consumer's local effect safely repeatable. Test the uncertain boundaries before expanding the pattern across every service.

The first version should include enough operational capability to answer whether work is progressing and how failed work resumes. That means visible event age, stable identifiers, a controlled quarantine path, and an owner for reconciliation. A minimal implementation can still have a complete recovery story.

After the first release, use observed failures to refine the design. Perhaps the payload needs a clearer version, the relay needs a better lease, or the consumer needs an external provider's idempotency key. Improve those concrete boundaries rather than adding architectural layers without a demonstrated need.

The practical success criterion is simple to state and demanding to prove. An accepted order leaves behind durable publication intent, retries do not silently multiply business effects, and operators can explain and recover the remaining uncertainty. That is the value the outbox should deliver.