Audit checklist

What this is

The procedure applied to every repository audited here. Three constraints shaped it.

It has to fit a working audit. Tier 1 is mandatory and scoped to fit a single audit cycle. Tier 2 items are conditional and fire only when a Tier 1 result triggers them. A checklist that cannot be completed is a checklist that gets selectively completed, which is the failure mode these audits exist to expose.

Every item has to be falsifiable by someone else. Each check names the artefact it produces. A reader with the repository and that artefact can confirm or overturn the finding without taking my word for anything.

The clean result is a result. An audit that finds nothing is published at the same length and with the same prominence as one that finds everything.

1. Admissibility gate

Run before any work begins. A repository that fails any of these is not audited and is not listed as a candidate. The reasoning behind the exclusions is in the ethics policy.

• A1Explicit claim. A specific performance claim in a durable, quotable artefact — README, paper, release notes, docs. The exact wording, file and commit are recorded. An implied claim, or one made only in a video or a social post, is not admissible.
• A2Not a first project. The author has other substantive public work, or the repository is institutionally authored.
• A3Not a minor. No indication the author is under 18. Pseudonymous authors are audited on the same terms as named ones, and no attempt is made to identify them.
• A4Reproducible in principle. Data is public, purchasable, or substitutable with a documented proxy. Where the claim depends on data nobody can obtain, that is the finding.
• A5Licence permits the work. Recorded.
• A6No conflict. Not a sponsor, not a sponsor's competitor, not a repository I have contributed to.

2. Reproduction gate

Before asking whether a result is right, establish whether it is obtainable. Some audits end here, and that ending is publishable.

• R1Pin the commit. A specific SHA, never a branch.
• R2Clean-environment install. Fresh container, documented steps only. Every undocumented step required is recorded.
• R3Run as instructed. Documented entry point, documented parameters, no fixes yet.
• R4Compare to the claim. The numeric delta is recorded whether or not it reproduces.
• R5Determinism. Three runs, variance recorded. Where the pipeline is stochastic, whether the claimed figure is a single run, a mean or a maximum. An unreported maximum-over-runs is a finding.
• R6Version drift. Whether dependencies are pinned, and the resolved versions used.

3. Classical backtest integrity Tier 1

Applies to every audit, LLM-agent audits included. An agent is still a backtest, and the classical defects do not stop applying because a model is involved.

Temporal integrity

• T1Timestamp semantics. For every input series: event time, exchange receipt, vendor publication, or bar close. Vendor-published-at is the only defensible basis for a decision, and it is the one most often missing.
• T2Decision/execution offset. A signal computed at t executes at t+1 or later, at a price actually available then. Same-bar close execution and missing shifts are flagged.
• T3Restatement and revision. Fundamentals and macro as first published, not as currently revised.
• T4Whole-sample transforms. Any scaler, normaliser, winsoriser, imputation or feature selection fitted before the split. The most common defect in ML trading repositories.
• T5Target leakage. No feature computed from a window containing its own label.
• T6Corporate actions. Split and dividend adjustments applied as-of, not retroactively.

Universe construction

• U1Survivorship. Delisted, merged and bankrupt names present for the period tested. A universe sourced from a current index membership list fails this.
• U2Point-in-time membership. Constituents as of each rebalance date.
• U3Selection provenance. A hand-picked universe over a period chosen after the fact is not a backtest. It is recorded as an illustration.
• U4Listing-date validity. No instrument trading before it existed.

Cost realism

• C1Explicit costs. Commission, exchange and clearing fees, compared against a real broker schedule for the instrument.
• C2Spread. Crossing it is a cost, and at the holding periods most of these repositories use it usually dominates everything else.
• C3Slippage and impact. Zero slippage is a finding, not a default.
• C4Financing and borrow. Short borrow, margin interest, futures roll, perp funding.
• C5Breakeven cost. The round-trip cost in basis points at which the claimed excess return goes to zero. Reported in every audit.
• C6Capacity. Position size against median traded volume over the period.

Selection and multiple testing

• S1Trial count. How many configurations, parameter sets, universes, date ranges and model variants were tried — derived from commit history, config files and notebooks. Where N is unrecoverable, that is itself a material finding.
• S2Deflated Sharpe Ratio. Computed fromN, sample length, skew and kurtosis. Raw and deflated both reported.
• S3Minimum backtest length. Compared against the actual backtest length.
• S4Probability of backtest overfitting.Where the repository exposes a trial matrix — a sweep, a grid search, an optimiser log — CSCV is run and PBO reported.
• S5Threshold. The multiple-testing adjusted significance threshold applied, and why.
• S6Out-of-sample honesty. Evidence that the holdout informed design — a fix committed after a disappointing result, a threshold tuned to the test set — is in-sample-by-iteration and is reported as such.

For S3, with N independent trials and γ the Euler–Mascheroni constant:

A backtest shorter than its minimum length cannot distinguish skill from selection, whatever its Sharpe ratio.

Statistical adequacy

• P1Sample size. Independent trades, not bars. Both reported.
• P2Concentration. Share of total P&L in the top 1%, 5% and 10% of trades. A strategy whose result rests on three trades has an n of three.
• P3Regime dependence. Non-overlapping subperiods reported separately.
• P4Benchmark. Buy-and-hold on the same instrument over the same window, cost-adjusted identically, plus a random-entry control with matched exposure and holding period. Excess over the correct baseline is the only number that matters.
• P5Metric definitions. Annualisation, risk-free rate, compounding convention, gross or net — recomputed independently. Definitional discrepancies are reported as errata, not as misconduct.

4. LLM agent contamination Tier 1

Applies wherever any component of the pipeline is a pretrained language model. This is the primary track.

Establish the window

• L1Model identification. Every model, by exact version string, taken from code and config rather than from the README. Any fine-tuning, and the corpus and date range it used.
• L2Cutoff determination. The vendor-stated pretraining cutoff for each model, with source and retrieval date. Where no cutoff is stated, the window is treated as unbounded — which makes every historical test in-window.
• L3Overlap map. Backtest period against each model's training window. Where the entire evaluation period sits inside the training window, the result is uninterpretable as evidence of reasoning, and that is the finding. Everything below quantifies how much.

The post-cutoff replication

• L4Matched out-of-window rerun. The unmodified pipeline re-run on a period strictly after the stated cutoff, matched on benchmark return, realised volatility and dispersion — with the match statistics published so a reader can judge whether the comparison is fair. In-window and out-of-window results side by side, with the decay in Sharpe and total return as percentages.
• L5Decay interpretation. Stated against the published reference range. Decay near zero is evidence for the repository and is reported with equal prominence.

Corroborating probes

At least two are run. They are cheap relative to L4, and they distinguish contamination from ordinary regime failure — which L4 on its own cannot do.

• L6Direct memorisation probe. Date-scoped factual questions about the backtest window — closing prices, direction over a period, event outcomes, rankings — scored against chance. Above-chance recall establishes that the information is present in the model, independent of whether the agent uses it.
• L7Lookahead-propensity conditioning. Each entity–date pair scored for recall probability, then tested for whether forecast skill concentrates on high-recall pairs. Skill present on high-recall pairs and absent on low-recall pairs is contamination, not reasoning. The effect should disappear post-cutoff.
• L8Counterfactual input perturbation. The inputs the agent claims to use are perturbed — a price sequence inverted, an earnings figure swapped, an indicator flipped — and the rate at which predictions change is reported. Published measurements find the worst agents leaving 82% of predictions unchanged under material input changes. Such an agent is not reading its inputs, and its backtest measures nothing.
• L9Entity anonymisation. Company identifiers stripped and the pipeline re-run — reported with its known confound. Removing the name also removes a distraction effect, which can exceed the look-ahead effect and invert the naive reading. Never reported as a standalone contamination test.

Harness leakage

The unglamorous failures, and the most common.

• L10Prompt window. Every prompt actually sent is logged and diffed. Whether the context contains data dated after the decision timestamp. The template is not read as evidence.
• L11Retrieval corpus scoping. Where RAG is used, whether the index is filtered to documents published before the decision timestamp. An unfiltered corpus over full history is look-ahead regardless of the model.
• L12Label leakage in the system prompt.Benchmark answers, evaluation labels or outcome hints in the system prompt or few-shot examples.
• L13Temperature and sampling. Temperature, seed, sample count, and whether the claimed figure is a single run, a mean or a best-of-n. An unreported best-of-n over a stochastic model is selection bias with an N the author never counted, and feeds back into S1.
• L14API drift. Whether the model version is pinned. A floating alias means the published result is not reproducible even by its author.

5. Conditional deep dives Tier 2

These fire only when triggered. Each names its trigger.

• X1Trial reconstruction from git history.Trigger: S1 unrecoverable.
• X2Data vendor cross-check.Trigger: R4 delta above 10%. Separates a code defect from a data defect.
• X3Cost-curve sweep.Trigger: C5 breakeven within 3× of realistic cost.
• X4Second-model replication.Trigger: L4 decay above 40%. Confirms the effect tracks the cutoff rather than the model.
• X5Position-level forensics.Trigger: P2 shows over 50% of P&L in the top 5% of trades. Those trades examined individually for data errors, stale quotes and unfillable prices.

6. Verdicts

Every completed audit carries exactly one.

7. What every audit must report

• The commit SHA audited, and the claim quoted verbatim.
• Environment specification and full run log.
• The breakeven cost.
• The correct-baseline comparison.
• Derived N, deflated Sharpe, and minimum backtest length against actual.
• For LLM audits: the overlap map, and the in-window / out-of-window table with regime-match statistics.
• Every checklist item marked not applicable, with the reason.
• The author notification date, the response deadline, and the author's response in full — or a statement that none was received.
• My own error bars: what this audit did not test, and what would overturn it.

Several of these are enforced at build time rather than by good intentions. An audit missing its breakeven cost, its baseline, its trial count or its notification record does not render — it fails the build.

8. How findings are written

Findings describe code and results. They do not describe intent.

Suspicion, inference and demonstrated fact are labelled distinctly, every time. The full rule, and why it is a hard one, is in the ethics policy.

9. References

1. Bailey, D. & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. SSRN 2460551.
2. Bailey, D., Borwein, J., López de Prado, M. & Zhu, Q. (2014).Pseudo-Mathematics and Financial Charlatanism. Notices of the AMS 61(5).
3. Harvey, C., Liu, Y. & Zhu, C. (2016). …and the Cross-Section of Expected Returns. Review of Financial Studies.
4. Gao, Z., Jiang, W. & Yan, Y. Detecting Lookahead Bias in LLM Forecasts. arXiv:2512.23847.
5. Profit Mirage: Revisiting Information Leakage in LLM-based Financial Agents. arXiv:2510.07920.
6. Glasserman, P. & Lin, C. (2023). Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis. arXiv:2309.17322.
7. Yan, Y., Tang, R., Gao, Z., Jiang, W. & Lu, Y. (2026). DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining. arXiv:2603.11838.

10. Amendment

This checklist is versioned. Changes are published with a dated changelog and a statement of which existing audits, if any, would reach a different verdict under the new version. Audits are never silently re-scored.