Implementing Self-Trade Prevention in a Matching Engine

When a trader's buy order matches against their own sell order, the result is a self-trade — also known as a wash trade. No economic value is exchanged. The trader is both buyer and seller. On regulated exchanges, this is prohibited because it artificially inflates volume and can be used for market manipulation.

This article walks through the implementation of self-trade prevention (STP) in MatchEngine, an open-source order matching engine written in Go. We added four configurable STP modes, each with different behavior when a self-trade is detected.

The Problem

Before this change, the engine had no concept of order ownership. Every order was anonymous:

type Order struct {
    ID        string
    Side      Side
    Type      OrderType
    Price     decimal.Decimal
    Quantity  decimal.Decimal
    Remaining decimal.Decimal
    Timestamp time.Time
}

If Alice placed a sell at 100 and then a buy at 100, the engine would happily match them. Alice would be both the maker and the taker. The trade would appear in the trade log, inflate the volume counter, and potentially trigger downstream systems (risk checks, fee calculations, market data feeds) — all for a transaction that moved nothing between parties.

The Solution: OwnerID + STP Modes

Step 1: Add Ownership

We added an OwnerID field to the Order struct:

type Order struct {
    ID        string
    OwnerID   string          // identifies the trader
    Side      Side
    Type      OrderType
    Price     decimal.Decimal
    Quantity  decimal.Decimal
    Remaining decimal.Decimal
    Timestamp time.Time
}

And to OrderRequest for the auto-ID API:

type OrderRequest struct {
    OwnerID  string
    Side     Side
    Type     OrderType
    Price    decimal.Decimal
    Quantity decimal.Decimal
}

// Builder method for fluent API
func (r OrderRequest) WithOwner(ownerID string) OrderRequest {
    r.OwnerID = ownerID
    return r
}

The OwnerID is a plain string. It could be a user ID, an account number, a firm identifier — whatever the caller uses to distinguish participants. The engine does not interpret it; it only checks equality.

Step 2: Define STP Modes

Different exchanges handle self-trades differently. We implemented the four most common modes:

type STPMode int

const (
    STPNone           STPMode = iota // disabled (default)
    STPCancelResting                  // cancel the resting order
    STPCancelIncoming                 // cancel the incoming order
    STPCancelBoth                     // cancel both
    STPDecrement                      // reduce both by overlap qty
)

Each mode answers the question: "When a self-trade is about to happen, what do we do with the two orders?"

Mode	Resting Order	Incoming Order	Trade Produced?
CancelResting	Removed from book	Continues matching	No
CancelIncoming	Stays in book	Cancelled entirely	No
CancelBoth	Removed from book	Cancelled entirely	No
Decrement	Reduced by overlap	Reduced by overlap	No

The mode is configured at the engine level:

e := engine.New(engine.WithSTPMode(model.STPCancelResting))

Step 3: Detection

Self-trade detection is a simple equality check:

func isSelfTrade(a, b *model.Order) bool {
    return a.OwnerID != "" && a.OwnerID == b.OwnerID
}

Two important details:

1. Empty OwnerID bypasses STP. If either order has no owner, they always match. This preserves backward compatibility — existing code that doesn't set OwnerID continues to work exactly as before.

2. The check is string equality. No normalization, no case folding. The caller is responsible for consistent owner identifiers. This is deliberate — the engine should not guess what "alice" vs "Alice" means in the caller's domain.

Step 4: Integration Into the Matching Loop

The STP check is inserted into the matching loop, right after the price check and before trade execution:

func (e *Engine) matchBuy(book *orderbook.OrderBook, order *model.Order) []model.Trade {
    var trades []model.Trade

    for !order.IsFilled() {
        bestAsk := book.BestAsk()
        if bestAsk == nil {
            break
        }
        if order.Type == model.Limit && bestAsk.Price.GreaterThan(order.Price) {
            break
        }

        // Self-trade prevention
        if e.stpMode != model.STPNone && isSelfTrade(order, bestAsk) {
            if e.handleSTP(book, order, bestAsk) {
                continue // resting removed, try next level
            }
            break // incoming cancelled
        }

        trade := executeTrade(order, bestAsk, bestAsk.Price)
        trades = append(trades, trade)
    }

    return trades
}

The handleSTP method returns a boolean: true means "continue matching" (the resting order was removed, try the next one), false means "stop" (the incoming order was cancelled).

This control flow is the key design decision. When a self-trade is detected at the best price, the engine doesn't just skip that one order — it applies a policy that may affect both orders and determines whether matching should continue.

Step 5: The STP Handler

func (e *Engine) handleSTP(book *orderbook.OrderBook, incoming, resting *model.Order) bool {
    switch e.stpMode {
    case model.STPCancelResting:
        book.RemoveOrder(resting.ID)
        delete(e.orderIndex, resting.ID)
        return true

    case model.STPCancelIncoming:
        incoming.Remaining = decimal.Zero
        return false

    case model.STPCancelBoth:
        book.RemoveOrder(resting.ID)
        delete(e.orderIndex, resting.ID)
        incoming.Remaining = decimal.Zero
        return false

    case model.STPDecrement:
        overlap := decimal.Min(incoming.Remaining, resting.Remaining)
        incoming.Remaining = incoming.Remaining.Sub(overlap)
        resting.Remaining = resting.Remaining.Sub(overlap)
        if resting.IsFilled() {
            book.RemoveOrder(resting.ID)
            delete(e.orderIndex, resting.ID)
        }
        if incoming.IsFilled() {
            return false
        }
        return true

    default:
        return true
    }
}

Let's trace through each mode with a concrete example.

Walkthrough: Alice Sells 5, Then Buys 7

Setup: Alice has a resting sell order for 5 units at price 100. Bob has a resting sell order for 3 units at price 100 (placed after Alice). Alice submits a buy for 7 units at price 100.

Book before:
  Asks: [alice: SELL 5 @ 100] [bob: SELL 3 @ 100]

Incoming: alice: BUY 7 @ 100

Mode: CancelResting

1. Best ask is Alice's sell (5 @ 100). Self-trade detected.
2. Alice's sell is removed from the book. Incoming buy continues (remaining: 7).
3. Best ask is now Bob's sell (3 @ 100). Different owner. Trade: 3 @ 100.
4. Buy has 4 remaining. No more asks. Buy rests in book.

Result: 1 trade (alice buys 3 from bob). Alice's sell cancelled. Alice's buy rests with 4 remaining.

Mode: CancelIncoming

1. Best ask is Alice's sell (5 @ 100). Self-trade detected.
2. Alice's buy is cancelled entirely (remaining set to 0).
3. Matching stops.

Result: 0 trades. Alice's sell stays in book. Alice's buy is gone.

Mode: CancelBoth

1. Best ask is Alice's sell (5 @ 100). Self-trade detected.
2. Alice's sell is removed. Alice's buy is cancelled.
3. Matching stops.

Result: 0 trades. Both orders gone. Bob's sell still in book.

Mode: Decrement

1. Best ask is Alice's sell (5 @ 100). Self-trade detected.
2. Overlap = min(7, 5) = 5. Both reduced by 5.
3. Alice's sell: 5 - 5 = 0 (filled, removed from book).
4. Alice's buy: 7 - 5 = 2 (continues matching).
5. Best ask is Bob's sell (3 @ 100). Different owner. Trade: 2 @ 100.
6. Buy is filled.

Result: 1 trade (alice buys 2 from bob). No self-trade produced. Alice's sell silently decremented to zero.

Design Decisions

Why Engine-Level, Not Per-Order?

The STP mode is set on the engine, not on individual orders. This is a simplification. Some exchanges allow per-order STP modes (e.g., one order uses CancelResting while another uses CancelIncoming). We chose engine-level for two reasons:

1. Simplicity. One mode for the entire engine is easier to reason about and test.
2. Consistency. Mixed modes within the same book can produce surprising behavior. If order A uses CancelResting and order B uses CancelIncoming, and they self-trade, which mode wins?

If per-order modes are needed later, the handleSTP method can be extended to read the mode from the order itself. The matching loop doesn't need to change.

Why Not Skip Instead of Cancel?

An alternative to cancelling is skipping — leave the resting order in the book and move to the next one. This sounds simpler but creates a problem: the skipped order is still at the best price. The next time the same owner submits a matching order, the same self-trade check fires again. The order becomes permanently unmatchable by its owner but still visible to everyone else.

Cancelling (or decrementing) is cleaner. It removes the conflict and lets the matching loop proceed to the next price level without leaving orphaned orders.

Why Empty OwnerID Bypasses STP?

Backward compatibility. Before this feature, no orders had an OwnerID. All existing code that creates orders without setting OwnerID should continue to work identically. Making empty-owner orders exempt from STP achieves this without any migration.

It also serves a practical purpose: anonymous or system-generated orders (e.g., liquidation orders) that should match against anything can simply omit the OwnerID.

Testing

Nine test cases cover the feature:

Test	What It Verifies
TestSTPDisabledByDefault	Self-trades happen when STP is off
TestSTPCancelResting	Resting cancelled, incoming matches next level
TestSTPCancelIncoming	Incoming cancelled, resting preserved
TestSTPCancelBoth	Both cancelled, book empty
TestSTPDecrement	Both reduced, no trade produced
TestSTPDecrementPartial	Decrement + match against different owner
TestSTPDifferentOwnersMatch	Different owners match normally with STP on
TestSTPEmptyOwnerIDAlwaysMatches	Empty owner bypasses STP
TestSTPWithAutoGeneratedIDs	STP works with SubmitRequest + WithOwner

The TestSTPDecrementPartial test is the most complex — it verifies that after decrementing against a same-owner resting order, the incoming order continues to match against a different owner's order at the same price level.

Usage

// Create engine with STP enabled
e := engine.New(
    engine.WithSTPMode(model.STPCancelResting),
    engine.WithIDPrefix("ORD-"),
)

// Using SubmitOrder (legacy API)
sell := model.NewLimitOrder("s1", model.Sell, price, qty)
sell.OwnerID = "alice"
e.SubmitOrder("BTC/USD", sell)

// Using SubmitRequest (recommended API)
req := model.NewLimitOrderRequest(model.Buy, price, qty).WithOwner("alice")
id, trades, err := e.SubmitRequest("BTC/USD", req)
// trades will be empty — self-trade prevented

Summary

Aspect	Detail
New types	STPMode enum, OwnerID field on Order/OrderRequest
Configuration	WithSTPMode(mode) engine option
Modes	CancelResting, CancelIncoming, CancelBoth, Decrement
Default	STPNone (disabled, backward compatible)
Detection	String equality on OwnerID
Bypass	Empty OwnerID always matches
Tests	9 new test cases
Files changed	model/stp.go (new), model/order.go, model/request.go, engine/engine.go

The matching algorithm gained exactly two new lines in the hot path: an isSelfTrade check and a handleSTP call. When STP is disabled (the default), the stpMode != STPNone check short-circuits immediately — zero overhead for engines that don't need it.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.5.0
Issue: #7 — Add self-trade prevention