Is your trading agent's backtest overfit? Checking the Deflated Sharpe Ratio with one API call

If you let an agent search for a trading strategy, it will find one with a great Sharpe ratio. That part is easy. The hard part is knowing if the edge is real or if the agent just tried enough variants that one of them got lucky.

Here is how to check it with one API call.

Why agents overfit

An agent that tunes a strategy usually loops. Change the lookback. Change the stop. Add a filter. Run the backtest again. Keep the best one.

Each run is a trial. If you run 200 trials on noise, the best one will still look good. The Sharpe you report is the max of 200 draws, and the max of 200 draws is always higher than a typical draw. Your agent doesn't know that unless you tell it.

The usual Sharpe ratio ignores all of this. It also assumes returns are roughly normal and that you have plenty of data. Short backtests with fat tails or negative skew make it look better than it is.

What the Deflated Sharpe Ratio does

The Deflated Sharpe Ratio comes from David Bailey and Marcos Lopez de Prado (2014). It starts from the Probabilistic Sharpe Ratio, which is the probability that the true Sharpe is above some benchmark, given your sample length, skew and kurtosis.

The deflated version raises that benchmark. The more trials you ran, the higher the Sharpe you would expect from luck alone. DSR asks how likely it is that your Sharpe beats that luck hurdle.

So it adjusts for four things:

• how many strategy variants you tried
• skew of the returns
• kurtosis (fat tails)
• how many periods of data you have

The output is a probability between 0 and 1. A common cutoff is 0.95.

The API call

I built a service called Agent Financial Advisor, and one of its tools is strategy_audit. If you send it an empty body it tells you what it needs:

curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
  -H "content-type: application/json" -d '{}'

Provide trades (with pnl) or at least 2 periodic returns.

The input schema (from GET /v1/tools) takes trades (each with pnl or return_pct), or returns as decimals, plus periods_per_year, n_trials, an optional trial_sharpe_variance and starting_equity.

Here are 24 monthly returns from a made-up strategy. First I say it was the only thing I tested:

curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
  -H "content-type: application/json" \
  -d '{"returns": [0.021,-0.012,0.034,0.008,-0.025,0.041,0.015,-0.006,0.027,0.011,-0.031,0.038,0.019,0.004,-0.014,0.029,0.022,-0.009,0.012,0.035,-0.018,0.024,0.007,0.016], "periods_per_year": 12, "n_trials": 1}'

The response (trimmed to the relevant fields):

{
  "red_flags": [],
  "return_stats": {
    "periods": 24,
    "annual_return_cagr": 0.1286,
    "annual_volatility": 0.0708,
    "sharpe": 1.751,
    "sortino": 3.604,
    "max_drawdown": 0.031,
    "skew": -0.411,
    "kurtosis": 2.169,
    "probabilistic_sharpe_ratio": 0.9838,
    "deflated_sharpe_ratio": 0.9838,
    "sharpe_hurdle_annualized": 0,
    "n_trials": 1
  },
  "verdict": "robust"
}

A Sharpe of 1.75 and a PSR of 0.98. With one trial the DSR equals the PSR. Looks great.

Now tell it the truth

Say the agent actually tried 200 parameter combos before it picked this one. Same returns, only n_trials changes. Response trimmed again:

curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
  -H "content-type: application/json" \
  -d '{"returns": [0.021,-0.012,0.034,0.008,-0.025,0.041,0.015,-0.006,0.027,0.011,-0.031,0.038,0.019,0.004,-0.014,0.029,0.022,-0.009,0.012,0.035,-0.018,0.024,0.007,0.016], "periods_per_year": 12, "n_trials": 200}'

{
  "red_flags": [
    "Deflated Sharpe Ratio 0.266 is below 0.95: after accounting for 200 trial(s), the Sharpe is not convincingly above what luck would produce."
  ],
  "return_stats": {
    "periods": 24,
    "sharpe": 1.751,
    "skew": -0.411,
    "kurtosis": 2.169,
    "probabilistic_sharpe_ratio": 0.9838,
    "deflated_sharpe_ratio": 0.2658,
    "sharpe_hurdle_annualized": 2.262,
    "n_trials": 200
  },
  "verdict": "promising_but_unproven",
  "recommendations": [
    "Validate out of sample (walk-forward or a held-out period) before adding capital.",
    "Model commissions, spread and slippage explicitly; small edges disappear first."
  ]
}

The Sharpe didn't move. The hurdle did. After 200 tries you would expect a Sharpe around 2.26 from luck alone, and 1.75 is under that. The DSR drops from 0.98 to 0.27.

That is the whole point. Same backtest, very different confidence, and the only change was being honest about how many things were tried.

How I use it in an agent

The important part is counting trials. I have the agent keep a counter every time it runs a backtest on a new variant, and it passes that number as n_trials. If it has the Sharpe of every trial, it can pass their variance as trial_sharpe_variance too.

Then the rule is simple. If deflated_sharpe_ratio is under 0.95 or red_flags is not empty, the strategy doesn't get capital. It goes to an out of sample test first. That step is realy cheap compared to losing money on a fluke.

24 months is also short. More data helps the PSR a lot, so if you have daily returns, send those with periods_per_year set to 252.

Try it

The tool list and schemas are at https://agent-financial-advisor.agent-financial-advisor.workers.dev/llms.txt. There is also an MCP endpoint at /mcp if your agent speaks MCP. You get 5 free calls a day per IP, or 25 a day with a free key.

Outputs are automated checks, not investment advice.