Backtest and oracle: where does this price come from? Two books read

Your strategy, replayed on past prices, shows a Sharpe ratio of 2.8, an excellent return-to-risk score. You switch the bot on with real money and it loses in the first week. These are illustrative numbers, not a memory: you have been there, or you will be. Either way, the figure that convinced you came from somewhere you never looked.

A backtest replays a strategy on past prices to see what it would have earned. It is only worth something if every price it uses really existed at the moment the strategy would have acted.

The Sharpe ratio scores the result: the average return, divided by volatility. The return is counted above the risk-free rate, what a safe investment pays. Volatility is how widely returns swing.

I have worked on the centralized side of this trade, meaning platforms run by a company. As a freelancer I write Python crypto trading bots that run around the clock. They handle a stop-loss, an order that cuts a losing position. They also handle a trailing stop, an exit threshold that follows the price up, and position sizing.

At Goin Invest I wrote trading algorithms in Go for managed mandates: running a client's portfolio under a contract. They connected to five exchanges (platforms where cryptocurrencies are traded), including Binance, Kraken, Bybit and OKEx.

The other side escaped me: DeFi, decentralized finance, where the program itself is the marketplace. So I read two books from Packt's ExpertEdge subscription: Quantitative Trading by Ernest P. Chan (Wiley, 2nd edition, 2021) and Building Full Stack DeFi Applications by Samuel Zhou (Packt, 2024).

The first teaches you to check that a backtest price is honest. The second shows what happens when a program trusts a price someone manufactured for it.

Look-ahead: when your backtest uses a price that does not exist yet

A comparison: you play heads or tails, but you are allowed to read the result before you bet. You win every time, and you have no talent at all.

Look-ahead bias is that cheat, when it creeps into a backtest without you meaning it. It means using information that did not exist yet at the time of the trade.

Chan gives an example. A rule such as "buy when the price is within 1% of the day's low" cannot be applied live. That low is only known at the close. The backtest has the whole history at hand and never complains.

A trading day: at 10 am the rule buys near the day's low, but that low is only known at the close 10 am: the rule buys "within 1% of the day's low" Close: the day's low finally becomes known day's low open 10 am close Information from the future: that is look-ahead

Diagram made for this article, the time of day is an illustration. In red: the decision made with information that does not exist yet. In green: the moment it becomes known.

Chan proposes a truncation test (lesson 15). The idea: rerun the strategy with the last days hidden, and check that it decides the same way.

1. You compute your strategy's positions, what it holds on each date, on the full history. I call that result A.
2. You drop the last N days (he mentions 10 to 100), rerun, and get B.
3. You cut the last N rows off A and compare with B.

For Chan, any difference proves the program uses the future: "the discrepancies in positions mean that you are inadvertently using the truncated part of the historical data."

Here is my version in Python, not the book's. It is mine, adapt it to your strategy. It is the cheapest test in the trade, and almost nobody writes it:

import numpy as np
# prices and positions are pandas Series or DataFrame

def test_truncation(strategy, prices, n=60):
    """Fails if the strategy uses data from after the day it decides."""
    positions_a = strategy(prices)            # full history
    positions_b = strategy(prices.iloc[:-n])  # without the last n days
    # Over the common period, positions must be identical.
    common = positions_a.iloc[:-n]
    assert len(common) == len(positions_b)
    # equal_nan: a rolling window leaves NaNs at the start, in the same places
    assert np.allclose(common.values, positions_b.values, equal_nan=True), \
        "look-ahead: the future leaked into the positions"

# The classic trap this test catches (my own example):
# a signal normalised by the mean and standard deviation of the whole history.
# Replace it with a rolling window, for example prices.rolling(50),
# and the test passes.

(The comment about global normalization is my own illustration, it is not in the book.)

Wire this test into your continuous integration pipeline, the one that runs your tests automatically on every change. Every time someone touches the strategy, the test rereads history with a veil over the last few days.

Chan does not talk about crypto, but the principle carries over. A market that never closes has no "daily close". The temptation is to build a daily candle: the open, high, low and close of one day. Then to decide inside it.

Careful, the truncation test will not catch this trap. The day's candle is the same in both passes, because you only cut whole days. The counter is to decide on yesterday's candle and act today. You shift the signal by one day (signal.shift(1) with pandas, the Python data-table library).

Too many settings, too little data, too much leverage

An honest backtest still has two enemies: too many parameters and too little data.

A parameter (or setting) is a number you choose, such as a threshold or a duration. Chan gives a clear rule: "I would not employ more than five parameters," entry and exit thresholds, holding period and moving-average window included (lesson 15).

The disease is called data-snooping: optimize on the past long enough and you tune your strategy to noise. To picture it: memorizing the answers of an old exam. You get 20 out of 20 on that one, and nothing is proved for the next.

The book's remedy is to split the history in two. You tune on the first part, the training set. Then you test on the second, which you have never looked at.

On the amount of data, Chan relies on the calculations of Bailey and coauthors (2012) on the minimum length of a backtest. The backtest Sharpe is the one you measure on the past. The true Sharpe is the one the strategy will have live. A data point is a closing price, one per day:

• A backtest Sharpe of 1 needs 681 data points to be confident that the true Sharpe is at least 0. That is 2.71 years of daily prices.
• A backtest Sharpe of 2 needs only 174.
• To be confident that the true Sharpe exceeds 1, you need a backtest Sharpe of at least 1.5. That takes 2,739 points, almost 11 years of daily data.

I would not carry those numbers over to other frequencies without redoing the calculation.

Then there is leverage, the size of your portfolio divided by your own capital. Chan presents the Kelly criterion: the leverage that maximizes long-run growth. Its formula is the average return above the risk-free rate, divided by the variance, the square of volatility.

For SPY, the fund that tracks the S&P 500, Kelly gives 2.528. In practice most traders take half of it, half-Kelly. The reason: "it is always safer to have a leverage lower than what the Kelly formula recommends" (lesson 28).

The safeguard I like best is lesson 29: "the leverage to use is always the smaller of the half-Kelly leverage and the maximum leverage obtained using the worst historical loss." Kelly assumes bell-curve returns, while crashes are far more frequent than that.

The S&P 500 lost 20.47% in a single day on October 19, 1987. If you can tolerate a 20% loss in one day, the maximum leverage is about 1, against 1.26 for half-Kelly. You keep 1.

For a crypto bot I cannot derive a number from this, since Chan provides none. The principle carries over: the worst day has to be computed on your own data.

DeFi: how to rig a price, and how to defend against it

Now the contract side. A smart contract is a program deployed on a blockchain, and nobody can change it afterwards. A blockchain is a shared ledger, kept by a network of computers rather than by a company.

The pool: a scale that tips

A traditional exchange keeps an order book: the list of buy and sell offers, which it matches. An AMM does without. Another way to see it: a scale with two pans. ETH (Ethereum's currency) on one side, USDT (a dollar-pegged token) on the other. The price is read off the imbalance: the emptier a pan gets, the more what it held is worth.

An AMM (Automated Market Maker) is that scale written in code. It is a smart contract that holds two tokens, two digital currencies, in a shared reserve: the pool.

The rule in Uniswap v2, the best-known AMM, is called the constant product. You multiply the reserve of A by the reserve of B, and the result, called K, does not change with trades. My example, without fees:

The pool before and after buying 1 ETH: 5 ETH and 10,000 USDT at a price of 2,000, then 4 ETH and 12,500 USDT at a price of 3,125 Pool before the buy Pool after the buy 5 ETH 10,000 USDT 4 ETH 12,500 USDT K = 5 × 10,000 = 50,000 K = 4 × 12,500 = 50,000 Price: 2,000 USDT Price: 3,125 USDT I buy 1 ETH I pay 2,500 USDT

In green, the pool's reserves (indicative heights, one scale per token). In red, the price after the buy. K has not moved.

I buy 1 ETH: only 4 remain in the pool. K must stay at 50,000, so the USDT reserve rises to 50,000 / 4 = 12,500. I paid the difference, 2,500 USDT, and each ETH now sells for 12,500 / 4 = 3,125.

A single buy moved the price from 2,000 to 3,125. That is price impact. The book gives the extreme case: with equal reserves, buying 90% of one token multiplies its price by 100 (lesson 25). Keep that in mind, because it is exactly what makes a pool's price manipulable.

The oracle: a price read from a single pool

A contract knows no price on its own. It asks an oracle, the service that feeds it a price. A made-up case: you value your house by asking the opinion of a single neighbor. If someone managed to convince that neighbor of an absurd price, your valuation is absurd too.

The lazy solution is the same: read the reserve ratio of a single pool. Zhou judges it bluntly: "this way is extremely dangerous" (lesson 84).

The attack: inflate the price, then borrow against it

The attack revolves around a loan. To borrow, you deposit an asset as collateral, meaning a guarantee. The borrowing cap depends on the value of that deposit. With a collateral rate of 80%, you can borrow at most 80% of the deposit's value.

It also takes funds to buy with. A transaction is an order sent to the chain. A flash loan is an uncollateralized loan that must be repaid within the same transaction. If repayment fails, the whole transaction is cancelled.

Zhou details the attack in four steps (lesson 98). The flash loan supplies the funds at the start and is repaid at the end, all within a single transaction:

1. The attacker buys the token, which inflates its price in the pool.
2. They deposit a little of it as collateral. The oracle reads the inflated price, so it believes that deposit is precious.
3. They borrow other assets, far beyond the deposit's real value.
4. They sell the token back.

The four-step attack on a single-pool oracle: flash loan received, buy that inflates the price, collateral deposit, big loan, sell back, repayment, all within one transaction A single transaction: everything goes through, or everything is cancelled Start: the flash loan provides funds 1 2 3 4 Buy the token Deposit a little of it as collateral Borrow other assets Sell the token back The pool price inflates: this is the rigged price The oracle reads the inflated price, the deposit looks precious Far beyond the deposit's real value End: the flash loan is repaid

In red, the steps that produce or use the rigged price. In green, the flash loan, received at the start and repaid at the end.

By how much must the price be inflated? With an 80% collateral rate, a price inflated by 25% is enough. The borrowing cap then equals the deposit's real value. Indeed, 80% of 125% makes 100%.

The book cites the Euler Finance attack of March 2023: $197 million. It is the largest known flash-loan attack at the time of writing.

The defense: an average over 720 seconds

To get a feel for it: to know the price of gas, the average price over the week is more reliable than the price of one minute. Faking a minute is easy. Faking a whole week costs far more.

The book's defense follows that idea: an averaged oracle. A block is the batch of transactions the chain validates at once, roughly every 12 seconds. At each block, each token pair adds its price multiplied by the elapsed time to a running total.

The oracle derives a TWAP (time-weighted average price) from it over a window, here 720 seconds. That is 60 blocks of 12 seconds. A manipulation fits inside a single block, so it weighs little in the average. For Zhou, this approach "doesn't rely on multiple oracle sources while keeping the price manipulation resistant" (lesson 100).

A one-block price spike of 12 seconds, and the 720-second average that barely moves price time Averaging window: 720 s 720 s average (TWAP): barely moves One block (12 s): the instant price shoots up Instant price

Durations to scale (720 s against 12 s), illustrative prices. In red, the one-block spike. In green, the average that dilutes it.

The book's test replays the attack. After the manipulation, the averaged oracle values BAR, a test token from the book, at 2.16 ETH. The old oracle values it at 723.47 ETH. The attacker's loan is refused with the reason ACCOUNT_UNHEALTHY (lesson 105). Against the old oracle, the same scenario earned the attacker over 100 ETH (lesson 99).

It is the best section of the book: you see the rigged price and the honest price side by side, drawn from the same pool.

A checklist: "where does this price come from?"

Here is what I take away from setting the two books face to face. The list is mine, it is in neither Chan nor Zhou, but every line rests on a case they describe:

1. At what moment did this price really exist? If your strategy or contract reads it before that moment, you have look-ahead (the day's low known at the close).
2. Who can move it, and with how much capital? If a single order is enough, it is not a price. It is the opinion of whoever pays (the pool going from 2,000 to 3,125).
3. How long must it hold before anyone relies on it? An average over a window costs more to distort than a snapshot (12 seconds against 720).
4. How many independent sources? A source that can be moved alone is a flaw, even a clean one.
5. Is it the price I will actually pay? The backtest price is not the execution price, and the pool's price impact shows it in three lines.
6. On which dataset did I tune and on which did I test? If it is the same one, the price proved nothing.

What the two books do not say about real-world crypto

Chan barely mentions crypto. In my extraction the word appears once, in a list of asset classes handled by a backtesting platform.

His examples are stocks, ETFs and futures, with one closing price per day. An ETF is a traded fund that tracks an index. A future is a contract to buy or sell at a fixed date. Nothing on a continuous market or on connecting to several exchanges. His industrialization targets an individual, with Excel links and orders sent in batches to a broker.

On the stop-loss, Chan's thesis is that the tool only works in a trend. First case: the gap, a price that jumps from one level to another without trading in between. The order fills much lower, and the stop merely locks in the loss. Second case: mean reversion, when a price drifts away and then returns toward its average. There, the stop makes you exit "at the worst possible time" (lesson 37).

I am not converted for crypto: a continuous market has fewer gaps than a stock that is closed overnight. Only a test on crypto data will settle it, and the book provides none.

Zhou has the opposite flaw: he is full of crypto, but his DeFi stops at the basic services: swapping, staking, lending and the oracle. His lending protocol is based on Aave v2, one of DeFi's major lending protocols. v3 is only presented, with isolation mode and E-mode, features "we will not implement in this course."

Liquidation is the seizure of the collateral of a borrower whose debt exceeds their cap. The book explains it with numbers. A liquidator who pays 1,000 USDT for an account in default leaves with 1,100 USDT of collateral, bonus included (lesson 86). A close factor caps the share of the debt that can be repaid in one transaction (lesson 94). The code of the liquidation function, however, is not published in the course: the author finds it too long. The health factor, Aave's indicator of a loan position's health, appears nowhere in the extracted lessons.

On security, the book shows the right oracle test, but the attack is played with a test account, not with a real flash loan. In the extracted lessons, two things are missing:

• fuzzing, tests with random inputs;
• a mainnet fork, a test against a copy of the real chain.

The code has aged too. It talks to the chain with Ethers v5, a JavaScript library since replaced by v6. It starts the front end from create-react-app, a React starter tool since deprecated.

Neither book, in the parts read, connects its two worlds. Nobody there says how to backtest a strategy whose prices are those of a pool. The price impact there depends on the order you place. That question is open, and these books do not address it.

Which one to read, depending on your profile

Quantitative Trading: 6.0 out of 10 in my reading note. Read it if you write trading bots and want to know whether they really win. Read it too if you size your positions by gut feeling. Read lessons 15, 28 to 31 and 37, skim the rest. The code is written three times, in MATLAB, Python and R, and the universe (US stocks, Excel links) has aged. Pass if you want crypto or around-the-clock trading: the book barely covers it.

Building Full Stack DeFi Applications: 5.8 out of 10 in my reading note. Read it if you come from finance or back-end work and want to see an AMM, a lending pool and an oracle from the inside. Read the AMM and the oracle, skim the lending part, skip the React front end. It is not a book for deploying to production: no audit, and the liquidation code is not published in the course.

If you only have time for one and your bots run on centralized exchanges, take Chan. If you need to understand why a protocol you integrate with reads a given price, take Zhou's oracle part, lessons 98 to 105.

Conclusion

I went looking for two books about two trades and found a single question asked twice. In Chan, the misleading price comes from your own code, which looked too far ahead. In Zhou, it comes from a third party, who paid just enough to move it. In both cases nobody lied: the number was exact, it simply came from a place nobody had questioned.

What strikes me is that the remedies look alike. The truncation test removes the future to see whether decisions change. The TWAP dilutes the instant to see whether the price changes. Either way, you do not trust a number until you have removed the thing that could cheat it. I have not found a book that applies this discipline to a bot trading on a DeFi pool. I would like to read someone who has done it with data.