If you let an agent search for a trading strategy, it will find one with a great Sharpe ratio. That part is easy. The hard part is knowing if the edge is real or if the agent just tried enough variants that one of them got lucky.
Here is how to check it with one API call.
Why agents overfit
An agent that tunes a strategy usually loops. Change the lookback. Change the stop. Add a filter. Run the backtest again. Keep the best one.
Each run is a trial. If you run 200 trials on noise, the best one will still look good. The Sharpe you report is the max of 200 draws, and the max of 200 draws is always higher than a typical draw. Your agent doesn't know that unless you tell it.
The usual Sharpe ratio ignores all of this. It also assumes returns are roughly normal and that you have plenty of data. Short backtests with fat tails or negative skew make it look better than it is.
What the Deflated Sharpe Ratio does
The Deflated Sharpe Ratio comes from David Bailey and Marcos Lopez de Prado (2014). It starts from the Probabilistic Sharpe Ratio, which is the probability that the true Sharpe is above some benchmark, given your sample length, skew and kurtosis.
The deflated version raises that benchmark. The more trials you ran, the higher the Sharpe you would expect from luck alone. DSR asks how likely it is that your Sharpe beats that luck hurdle.
So it adjusts for four things:
- how many strategy variants you tried
- skew of the returns
- kurtosis (fat tails)
- how many periods of data you have
The output is a probability between 0 and 1. A common cutoff is 0.95.
The API call
I built a service called Agent Financial Advisor, and one of its tools is strategy_audit. If you send it an empty body it tells you what it needs:
curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
-H "content-type: application/json" -d '{}'
Provide trades (with pnl) or at least 2 periodic returns.
The input schema (from GET /v1/tools) takes trades (each with pnl or return_pct), or returns as decimals, plus periods_per_year, n_trials, an optional trial_sharpe_variance and starting_equity.
Here are 24 monthly returns from a made-up strategy. First I say it was the only thing I tested:
curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
-H "content-type: application/json" \
-d '{"returns": [0.021,-0.012,0.034,0.008,-0.025,0.041,0.015,-0.006,0.027,0.011,-0.031,0.038,0.019,0.004,-0.014,0.029,0.022,-0.009,0.012,0.035,-0.018,0.024,0.007,0.016], "periods_per_year": 12, "n_trials": 1}'
The response (trimmed to the relevant fields):
{
"red_flags": [],
"return_stats": {
"periods": 24,
"annual_return_cagr": 0.1286,
"annual_volatility": 0.0708,
"sharpe": 1.751,
"sortino": 3.604,
"max_drawdown": 0.031,
"skew": -0.411,
"kurtosis": 2.169,
"probabilistic_sharpe_ratio": 0.9838,
"deflated_sharpe_ratio": 0.9838,
"sharpe_hurdle_annualized": 0,
"n_trials": 1
},
"verdict": "robust"
}
A Sharpe of 1.75 and a PSR of 0.98. With one trial the DSR equals the PSR. Looks great.
Now tell it the truth
Say the agent actually tried 200 parameter combos before it picked this one. Same returns, only n_trials changes. Response trimmed again:
curl -X POST https://agent-financial-advisor.agent-financial-advisor.workers.dev/v1/tools/strategy_audit \
-H "content-type: application/json" \
-d '{"returns": [0.021,-0.012,0.034,0.008,-0.025,0.041,0.015,-0.006,0.027,0.011,-0.031,0.038,0.019,0.004,-0.014,0.029,0.022,-0.009,0.012,0.035,-0.018,0.024,0.007,0.016], "periods_per_year": 12, "n_trials": 200}'
{
"red_flags": [
"Deflated Sharpe Ratio 0.266 is below 0.95: after accounting for 200 trial(s), the Sharpe is not convincingly above what luck would produce."
],
"return_stats": {
"periods": 24,
"sharpe": 1.751,
"skew": -0.411,
"kurtosis": 2.169,
"probabilistic_sharpe_ratio": 0.9838,
"deflated_sharpe_ratio": 0.2658,
"sharpe_hurdle_annualized": 2.262,
"n_trials": 200
},
"verdict": "promising_but_unproven",
"recommendations": [
"Validate out of sample (walk-forward or a held-out period) before adding capital.",
"Model commissions, spread and slippage explicitly; small edges disappear first."
]
}
The Sharpe didn't move. The hurdle did. After 200 tries you would expect a Sharpe around 2.26 from luck alone, and 1.75 is under that. The DSR drops from 0.98 to 0.27.
That is the whole point. Same backtest, very different confidence, and the only change was being honest about how many things were tried.
How I use it in an agent
The important part is counting trials. I have the agent keep a counter every time it runs a backtest on a new variant, and it passes that number as n_trials. If it has the Sharpe of every trial, it can pass their variance as trial_sharpe_variance too.
Then the rule is simple. If deflated_sharpe_ratio is under 0.95 or red_flags is not empty, the strategy doesn't get capital. It goes to an out of sample test first. That step is realy cheap compared to losing money on a fluke.
24 months is also short. More data helps the PSR a lot, so if you have daily returns, send those with periods_per_year set to 252.
Try it
The tool list and schemas are at https://agent-financial-advisor.agent-financial-advisor.workers.dev/llms.txt. There is also an MCP endpoint at /mcp if your agent speaks MCP. You get 5 free calls a day per IP, or 25 a day with a free key.
Outputs are automated checks, not investment advice.
Top comments (5)
What does the API assume when trial_sharpe_variance is omitted? The winning return series alone does not show how dispersed the discarded trials were. I would include that assumption in the response alongside the hurdle.
There is also a distinction between 200 runs and 200 independent trials when adjacent parameter settings produce similar returns. Bailey and López de Prado discuss that dependence in the original paper: davidhbailey.com/dhbpapers/deflate... . Keeping the raw count and the independence assumption explicit would make the audit easier to reproduce.
If you leave it out, the API uses the sampling variance of the Sharpe estimate for your own return series, (1 − skew·SR + (kurtosis − 1)/4 · SR²) / (T − 1), as a stand-in for how much Sharpe varied across your trials. Real trials usually spread more widely than that, so the default makes the luck benchmark lower and the DSR more generous. If you have the Sharpe of each variant you tested, pass their variance in trial_sharpe_variance and the result tightens.
The part I would pin down before trusting a DSR from an API is what counts as a trial. In a parameter grid, neighbouring settings are nearly perfectly correlated, so 200 runs might be worth 20 independent ones — and the deflation then over-corrects, which is its own kind of wrong.
How does your service count trials: every call that returns a number, or does it estimate an effective N from the correlation between runs? And do abandoned runs that never produced a result get counted?
Agreed, that's the input that matters most. Count every variant you evaluated on the same data before choosing this one: each parameter set, universe, entry or exit rule tweak and lookback, including the ones you threw away. Pass that as n_trials. If you leave it out, the API assumes 1 and adds a note to the response saying the DSR will be too generous. When variants are highly correlated (say, a 20-day vs. 21-day lookback), counting effective independent trials is fairer, but a raw count is the safer default.
Both land, and the second one is the one I was dodging.
The count I would pass is the full evaluated set — every parameter set, universe, lookback and rule tweak I ran before settling on this version, discarded ones included — not the number of runs that survived. If I hand over only the surviving series, n_trials=1 is exactly what the API assumes, and the note it appends is the correct reading of my result.
On the default: my worry is the other direction. The fallback variance of a single series can come out tighter than the actual spread across my trials, so it pushes the same way as undercounting. The output worth printing is both numbers side by side — default variance and empirical trial variance — and how far the DSR moves between them.