Most backtests fail not because the idea behind them is bad, but because the test is dishonest — and the dishonesty is detectable. The twelve tests below are the standard we hold our own strategies to before anything leaves the research desk. We publish them for two reasons. First, a due-diligence standard you have to take on faith is a contradiction in terms. Second, we would rather be judged against a written standard than trusted on charm.
If you find a flaw in the standard, tell us. Serious critique improves it, and the changelog at the bottom of this page records every change. That is what publishing a standard is for.
How to read each test
Every test has three parts: what it tests, what a failure looks like, and the evidence it needs. The third part matters most, and it is the one most checklists omit. Some failures are visible in a returns series; others are invisible without the strategy’s rules, and a few are invisible without the code and the data pipeline. A checklist that pretends everything is verifiable from an equity curve is itself a small act of dishonesty — so ours says, test by test, what you cannot conclude with the material in front of you.
The twelve tests
01
Out-of-sample integrity
- What it tests
- Whether the result was demonstrated on data the strategy had no opportunity to learn from — a holdout period that was genuinely held out.
- What a failure looks like
- There is no holdout at all; or the ‘out-of-sample’ window was consulted repeatedly during development until it stopped being out of sample. A holdout you peeked at is an in-sample set with better marketing. The cleanest evidence in the other direction is post-publication data: a rule published years ago, evaluated on everything that came after, cannot have been fitted to it.
- Evidence it needs
- The dates are visible in a returns series, but whether the holdout was truly untouched is a fact about the research process. It can only be established from the protocol — or by asking, and listening to how the answer is given.
02
Walk-forward validation
- What it tests
- Whether parameters were estimated only on data available at each point in time, rolled forward — rather than fitted once across the whole history and then presented as if they had been tradeable all along.
- What a failure looks like
- A single full-history fit: parameters chosen with knowledge of 2026 applied retroactively to 2003. The resulting curve describes a strategy nobody could have run.
- Evidence it needs
- Requires the rules and the estimation protocol. A returns series cannot tell you when the parameters were chosen.
03
Transaction-cost & slippage realism
- What it tests
- Whether the full friction stack is modelled at levels realistic for the instruments and the intended size: commissions, spreads, slippage and market impact — and, wherever leverage or short positions appear, financing costs, borrow fees and roll costs. The treatment of cash (does an uninvested balance earn a rate, and is a levered position charged one?) should be stated, not assumed.
- What a failure looks like
- Zero or token costs; fills at mid-price; leverage that finances itself for free; shorts with no borrow. High-turnover strategies are the canary: an edge that dies when honest costs are applied was never an edge — it was a rebate for ignoring reality.
- Evidence it needs
- Verifiable from the returns level if the cost assumptions are disclosed alongside a trade list or turnover figure: assumed cost × turnover is an arithmetic check anyone can run. Without disclosed assumptions, the test is unverifiable — which is itself a finding.
04
Capacity & ticket-size honesty
- What it tests
- Whether the result would survive at the size the reader is actually being asked to imagine. A backtest is a claim about the past of a small amount of money; capacity asks whether the claim scales.
- What a failure looks like
- Signals harvested from illiquid instruments an institutional ticket would move; turnover incompatible with the mandate’s size; performance concentrated where the intended capital could never actually trade.
- Evidence it needs
- Partially checkable from a trade list plus liquidity data. Fully checkable only against a stated intended size — so the honest version of this test begins with the manager naming one.
05
Survivorship bias
- What it tests
- Whether the dataset includes the dead: delisted stocks, merged funds, discontinued contracts — the ones that would have been in the portfolio at the time and are missing from the database now.
- What a failure looks like
- Today’s index members backfilled through history, so the portfolio only ever contains companies we now know survived. The resulting edge is hindsight wearing a lab coat.
- Evidence it needs
- Invisible without the data pipeline. Neither a returns series nor the rules can prove the universe was point-in-time; only the data construction can.
06
Look-ahead bias
- What it tests
- Whether every signal used only information knowable at the moment it was acted on — data as published, not as later revised; prices that were actually tradeable when the signal fired.
- What a failure looks like
- Signals computed on restated fundamentals or final revised prints; a month-end signal executed at the very close whose value it needed to compute. Each instance is small; compounded across decades they manufacture an edge from a time machine.
- Evidence it needs
- Invisible at the returns level, and largely invisible even with the rules on paper. Reliably established only in the code and the data handling — which is why independent re-implementation exists.
07
Selection bias & multiple-testing accounting
- What it tests
- How many variants died so this one could live. Every research process tries things; an honest one counts them and adjusts its confidence accordingly.
- What a failure looks like
- Nobody can state how many strategies, universes, parameter sets and start dates were tried; significance is computed for the sole survivor as if it were the only candidate ever considered. Run a hundred random strategies and the best will look brilliant — the question is never how good the winner looks, but how large the tournament was.
- Evidence it needs
- A returns series can raise suspicion — results too good for the track length are a statistical tell — but the true trial count lives in the research log. It can only be disclosed, not detected.
08
Parameter sensitivity
- What it tests
- Whether neighbouring parameter values produce recognisably similar results — whether the strategy sits on a plateau or on a spike.
- What a failure looks like
- A performance cliff: the ten-month average works and the nine- and eleven-month versions die. A genuine effect is rarely that precise; a curve-fit almost always is.
- Evidence it needs
- Requires the rules, so the perturbations can be run. From returns alone, the shape of the parameter surface is unknowable.
09
Regime robustness
- What it tests
- Whether the edge survives across rate regimes, volatility regimes and crises — and whether the result depends on where the chart begins.
- What a failure looks like
- The entire edge earned in one lucky regime and presented as unconditional; a start date chosen just after a drawdown the strategy would not have survived. Moving the start date is the cheapest robustness test in finance, which is exactly why cherry-picked charts avoid it.
- Evidence it needs
- Substantially checkable from a returns series: sub-period analysis, regime tables and start-date shifts need nothing but the data and the willingness.
10
Data quality
- What it tests
- Whether the data is point-in-time, correctly adjusted (splits, dividends, and total-return where total-return is claimed), with gaps handled explicitly — and, where an instrument’s history is extended before its launch, whether the proxy is named and its differences acknowledged.
- What a failure looks like
- Index histories silently spliced where an instrument did not yet exist; adjustment errors quietly manufacturing returns; ‘price return’ benchmarked against ‘total return’ as if they were the same thing.
- Evidence it needs
- Needs the pipeline. Some symptoms surface in a returns series — impossible jumps, suspicious smoothness — but the construction of the dataset can only be audited at the source.
11
Statistical significance versus track length
- What it tests
- Whether the track is long enough to distinguish edge from luck at the Sharpe ratio being claimed. Modest edges need decades to prove; short tracks can only prove immodest ones.
- What a failure looks like
- A three-year track with a middling Sharpe presented as demonstration rather than anecdote; no confidence interval, no standard error, no acknowledgement that the number is an estimate at all.
- Evidence it needs
- The one test always fully checkable from a returns series alone. If a presenter of results has not run it, that too is information.
12
Reproducibility
- What it tests
- Whether an independent party, given the stated rules, could reproduce the claimed results — at minimum in structure, ideally in figures.
- What a failure looks like
- ‘Proprietary’ deployed as the answer to every methodological question, including the ones that reveal no secret; results that cannot be reconstructed even in outline. Protecting an edge is legitimate; a claim that cannot be examined by anyone, ever, under any terms, is not evidence — it is an anecdote with production values.
- Evidence it needs
- Requires the rules, under NDA where necessary. This is the gold standard of the audit tier system for a reason: reproduction is the only test that verifies all the others at once.
The standard is self-binding
Every strategy that leaves the Qnology research desk must survive these same twelve tests first. We do not publish a standard we exempt ourselves from — and where our own published research falls short of it, the standard obliges us to say so and fix it, in public, in the changelog. If you ever find our own material failing a test on this page, we want to hear about it more than anyone.
What this standard is not
Passing all twelve tests does not make a strategy good, and it does not make its future resemble its past. A clean backtest is a necessary condition for taking a strategy seriously — never a sufficient one, and never a promise. The standard addresses testing integrity only. Whether anyone should invest in anything is a different question, answered by different people: regulated advisers assessing suitability, which is not what we do and not what this page is.
Version & changelog
v1.0
2026-09-03
| Version | Changes |
|---|
| v1.0 — 2026-09-03 | Initial publication. Twelve tests, evidence boundaries, self-binding clause. |
The working checklist →
Qnology AI Ltd · The Qnology Standard v1.0 · Educational material — not investment advice · qnology.ai
