# ORION Backtest & Validation Log

> **Repository reproducibility audit — 2026-07-25:** The 2022 negative result remains in this
> log and is not weakened or replaced. The original AACT snapshots and one-off scripts are not
> present in the current repositories, so the entry is classified as a reported historical
> point-in-time reconstruction rather than a presently independently reproducible run.

**Purpose.** A running, dated index of every time ORION has tested one of its own formulas
against real historical outcomes — successful or not. The 2022 Clinical Risk backtest
(`docs/CASE_STUDY_2022_BACKTEST.md`) was originally a one-time case study; this log exists so
that kind of check becomes a recurring, disclosed practice instead of a single document that
ages out of relevance. **Every entry stays here permanently, in order, even a negative one** —
this index is never edited to remove or soften a past result, only appended to.

**What belongs here.** A test of whether a live ORION formula's *output* correlates with a
*real, historical, verifiable outcome* — trial terminations, FDA decisions, price moves, etc.
**What does not belong here:** ordinary code review, unit tests confirming an implementation
matches its own spec (that's correctness, not predictive validity), or manual QA. Those are
tracked in commit history and `test/`, not this log — conflating "we checked the arithmetic is
right" with "we checked the score predicts anything real" would undermine the entire point of
keeping this list honest.

---

## Log

| Date | What was tested | Sample size | Result | Full writeup |
|---|---|---|---|---|
| 2022 (published 2026) | ORION Clinical Risk™ vs. real 1-year-forward clinical trial termination, tested 5 ways | Up to 83,000 trials / 1,806 companies | **Negative** — r = 0.055, no meaningful correlation found across any of the 5 hypotheses tested | [`CASE_STUDY_2022_BACKTEST.md`](CASE_STUDY_2022_BACKTEST.md) |

*(No further entries yet. The next one goes here, at the top, when the next real backtest — e.g.
Regulatory Risk's calendar-proximity constants, or Buy Score's weighting once enough evaluated
predictions exist — is actually run. See §14 of `ORION_RISK_SCORE_SPEC.md` and §6 of
`ORION_BUY_SCORE_SPEC.md` for the validation work already identified as owed but not yet done —
those are the most likely candidates for entry #2.)*

---

## Why this list will sometimes only have negative or null results on it

Most of ORION's formulas are new enough, or cover categories added recently enough (Regulatory
Risk and Cash Burn Risk both shipped 2026-07-13/14), that there hasn't been time to accumulate
enough real evaluated outcomes to test them meaningfully yet. That is stated plainly rather than
implied to be settled — see the Track Record page (`orion-accuracy.html`) for the live,
continuously-updating evaluated-prediction count this kind of test depends on, and the 30-
evaluated-prediction threshold below which a result is treated as not yet statistically
meaningful.