When the Engine Stops: Recovery Without Losing the Market's State

A matching engine restarts. Bids and asks reappear. The process is healthy.

There is still a harder question to answer: did it recover the same market?

Two books can show identical depth while disagreeing about who was first at a price. A restarted engine can remember a fill that the account ledger has forgotten. A client can be waiting for a cancellation that already took effect.

Recovery has to reconcile these histories before new orders make them harder to untangle.

Choose an architecture before explaining its recovery

Harbor, our fictional exchange, trades a fictional instrument called BAG in whole units at one-dollar price increments. An order book records buy and sell instructions waiting to trade. The matching engine selects compatible orders and records their executions, or fills. Any unfilled quantity allowed to remain in the book is a working remainder.

Harbor’s orders are fully displayed and belong to different owners. Better prices have priority; earlier orders win at the same price. Execution uses the resting order’s price. A buy limit is the maximum payable price; a sell limit is the minimum acceptable price. Ordinary orders are good till canceled (GTC), so an eligible remainder can keep working under Harbor’s rules. Auctions, external routing, fees, and settlement are outside the example.

Use this recovery design for Harbor: it records admitted commands in an ordered journal, a durable history of instructions such as new orders and cancellations. Replicas are separate copies of that history and the processing state. A command is committed when the replication protocol has made it part of the authoritative recoverable history; applied means the matching process has carried out its effect on the book.

A snapshot is a saved copy of the application state at a known applied position. Replay restores a snapshot and processes later committed commands again. Deterministic processing means the same starting state and ordered inputs produce the same results. That lets a replacement process reconstruct the book rather than guess from the last screen a customer saw.

Assume three replicas and a protocol requiring a durable majority before a command is applied or a definitive business result is published. This example is intended to survive a single replica’s loss, with the other required storage and protocol assumptions holding. It does not promise survival when every durable copy is destroyed.

Aeron Cluster documents this general arrangement: a leader sequences a replicated log, services consume after a majority has recorded it, and snapshots plus subsequent log replay support recovery. It is an example of an available design, not evidence that Coinbase or Nasdaq uses Aeron internally. Aeron Cluster architecture

Martin Fowler’s 2011 account of LMAX describes another relevant implementation: durable input journaling before business processing and reconstruction from snapshots plus later inputs. Treat it as a historical architectural account, not a current performance specification. The LMAX Architecture

A log of trades leaves too much out

Harbor’s journal records commands, including new orders, amendments, and cancellations. Starting with the same state, the matching rules reproduce their results.

A list containing only trades cannot do that job. It omits the seller who joined the queue but never traded, the amendment that changed priority, and the cancellation that removed an unfilled remainder. Those events change what a later incoming order can do.

Another architecture could persist complete resulting state changes rather than replay inputs. That requires its own completeness and atomicity rules. Calling both things an “event log” can conceal which information is actually recoverable.

For Harbor, retain the inputs needed to reproduce each decision. That includes applicable rule and configuration changes. If expiry affects an order, its timing decision must also be recoverable. Aeron documents logged timer events and snapshot handling as one concrete solution; Harbor would need its own specified equivalent if we added expiring orders. Aeron cluster timers

The snapshot needs the queue

Our snapshot at command position 103 contains the book below. Each sequence number is a processing position on this market, with lower numbers coming first. Buy orders are bids, sell orders are asks, and live units are the quantity still available to trade:

Order	Side	Price	Live units	Priority sequence
B1	Buy	$99	6	100
S1	Sell	$100	4	101
S2	Sell	$100	3	102
S3	Sell	$101	5	103

Saving “seven offered at $100” loses the S1-before-S2 queue. Restoring those seven units in the opposite order would give the next buyer the same price and total quantity while changing which seller traded.

Harbor’s complete snapshot therefore includes order identity, owner association, side, price, live quantity, priority, accumulated execution state, and the journal position. It also includes the identifier counters, request-deduplication results, and output-recovery information required by this design. Configuration is pinned so replay uses the same rules.

That is our proposed snapshot contract. Public depth is a smaller view of the system and cannot substitute for it.

Stop just before the cancellation takes effect

After snapshot 103, Harbor commits these commands. M1 is the new buy order, S4 is a later sell order, and C1 is a request to cancel M1’s unfilled quantity:

Position	Command	Deterministic effect
104	M1 buys ten, limit $100, GTC	Buy seven at $100; rest three at $100
105	S4 sells two, limit $100	Trade two against M1; M1 has one left
106	C1 cancels M1	Remove the remaining one

Now crash after 106 is durably committed but before the matching process applies it. The client has received no final cancellation result.

Figure 5.1. The latest complete snapshot is through command 103. The failed process has applied 104 and 105. The authoritative committed journal also contains 106. Recovery must restore the snapshot and process the committed tail through 106.

Recovery loads snapshot 103 and replays committed commands 104 through 106. M1 finishes with nine units executed, one canceled, and zero working. S1 and S2 are gone. S4 is fully filled. B1 remains six bid at $99; S3 remains five offered at $101.

The crash delays the client’s knowledge. It does not authorize recovery to place C1 before S4 or forget C1 because its result was never delivered.

If the committed history instead ended at 105 and C1 existed only in a failed gateway’s memory, the conclusion would differ. The operator must establish the actual boundary rather than fill the gap with the customer’s intent.

Figure 5.2. Snapshot 103 preserves the original order book and priority. Replay 104 buys seven and rests three; replay 105 buys two more and leaves one working; replay 106 cancels that unit. The recovered book retains B1 buy six at $99 and S3 sell five at $101.

The snapshot is captured consistently with applied application state. The arrows show recovery responsibilities, not a claim that writing a journal automatically makes every downstream consumer consistent.

Replaying a fill must not buy the asset twice

Command 104 generates two executions. Command 105 generates another. Some reports may have reached clients or ledgers before the crash.

Harbor gives each resulting event a stable identity derived from the market’s history: for example, 104.1, 104.2, and 105.1, scoped to this journal’s unique identity. Replay regenerates those identities rather than allocating three unrelated trades.

Consumers retain enough state to recognize repeated events and apply each economic effect once. Delivery recovery resends or retrieves the reports the client missed. Rebuilding engine state and recovering report delivery are separate operations, even when both start from the same history.

The design also retains request identities and results for a defined retry window. A committed command can lose its response; a retry must not silently become a second business instruction. Raft’s authors discuss precisely this commit-before-response ambiguity and a serial-number/result approach to deduplication. Raft paper, section 8

Our chosen identity scheme is illustrative. A real protocol may scope identifiers by account, session, market, or trading day. A bare number is not automatically unique forever.

Establish one authority

A replacement process cannot safely publish while a former leader continues producing externally accepted results.

Harbor’s recovery procedure therefore establishes the current leader, the process authorized to order new work, and blocks the former leader from publishing or admitting conflicting work. Blocking the former process’s authority is called fencing. The output path honors the current authority as well. Electing a leader inside a replicated core does not automatically fence an unrelated gateway or ledger publisher.

Raft uses terms to recognize stale leaders within its protocol. Harbor must extend the authority boundary to the components that can create outside effects. This is a design requirement inferred for our system, not a claim about a particular exchange’s failover implementation. Raft paper, section 5

Nor should an operator select the machine with the longest local journal and declare it correct. A locally written tail may not belong to the committed history. The replication protocol defines the recoverable boundary.

The client’s book needs a different repair

Coinbase documents a client-side procedure that buffers full-channel updates, obtains a book snapshot, and applies messages after the snapshot’s sequence. That reconstructs a participant’s market-data copy. It does not document Coinbase’s internal engine backup format or prove a customer’s private execution history. Coinbase full-channel reconstruction

Harbor must separately reconcile its engine state, order and execution reports, gateway views, and relevant risk and ledger consumers. An accurate public ask does not establish whether M1 bought nine units.

Reconciliation also does not complete clearing or settlement. For a US securities comparison, those functions determine obligations and transfer securities and cash through their own infrastructure. DTCC clearing and settlement services

Before Harbor reopens

For this illustrative design, operators need evidence that:

• One authority can admit work and publish accepted results.
• A compatible snapshot and complete committed journal tail reproduce the expected state.
• Order quantities, priority, and stable identifiers survive replay.
• Missing reports can be recovered without duplicate economic postings.
• Open orders, executions, risk reservations, and ledger consumers agree at identified recovery positions.
• The market’s status and order-retention policy are explicit before intake resumes.

If Harbor cannot establish that history, restarting matching adds new activity to an unresolved state. It should remain stopped while the evidence is reconciled. A green process indicator is only the beginning of the recovery check.