# Case Study: What Happens When You Actually Backtest Your Own Model

> **Reproducibility status (audited 2026-07-25):** This negative result is preserved exactly
> as reported. It used two historical AACT snapshots and was designed as a point-in-time
> reconstruction, but those snapshot artifacts and the one-off analysis scripts are not
> committed in the current repositories. It is therefore **reported historical evidence, not
> independently reproducible from this repository today**. ORION will not upgrade that status
> until the hashed source artifacts and executable run are available.

**For institutional evaluators considering ORION.** This is the honest version of the case study every research-tool vendor should be able to produce and almost none do.

---

## The claim most vendors make

"Our model works." Usually backed by a hand-picked example where it happened to be right, and no visible methodology for how that example was chosen.

## What we did instead

We took ORION's own live scoring engine — the exact code running in production today, unmodified — and tested it against **real historical ClinicalTrials.gov data**, reconstructed as it actually looked on specific past dates, using AACT (the Clinical Trials Transformation Initiative's public archive at Duke University). Not today's data with hindsight. Not a cherry-picked example. The real state of the world, on a real past date, checked against what actually happened afterward.

**Five separate hypotheses, escalating in sample size:**

| # | What we tested | Sample size | Result |
|---|---|---|---|
| 1 | 5 named companies with real, sourced, dated 2022 FDA decisions (Immunocore, Akebia, Amylyx, Provention Bio, Y-mAbs) | 5 | No clean separation — our highest-scored company of the five received a CRL; our lowest-scored was approved three weeks later |
| 2 | Every industry-sponsored company with ≥3 trials, systematic sample | 1,806 companies | r = 0.055 — no meaningful correlation between our Clinical Risk score and real 1-year-forward trial termination |
| 3 | Regression-recalibrated weights, evaluated on a held-out test split | 1,806 companies, 70/30 split | The recalibrated weights performed *worse* on data they hadn't seen — ruling out "wrong weights" as the explanation |
| 4 | Trial recruiting duration as a stall proxy | 59,821 trials | r = 0.0005 — essentially zero |
| 5 | Completion-date revisions as a proxy | 82,935 trials | A real, if small, effect — but the *opposite* direction from the intuitive hypothesis |

## What we found

**Our Clinical Risk formula, as it exists in production today, does not meaningfully predict which companies' clinical trials will terminate over the following year.** We tested this five different ways, at sample sizes up to 83,000, and the honest answer held up every time.

We are publishing this. Not because we're required to. Because it's true, and because a fund that finds this out from a slide deck later instead of from us is a fund that stops trusting anything else we say.

## What we did about it

1. **We did not ship the recalibrated weights** — a worse-performing, "regression-calibrated"-sounding formula would have been a strictly worse product wearing a more sophisticated label.
2. **We did not remove the honest disclosure already in our methodology docs** ("reasoned, not backtested") — this backtest confirmed it was accurate, not just cautious.
3. **We redirected engineering effort.** The one place a real signal showed up — records that get actively revised behave differently from records that go quiet — pointed toward a genuinely different class of signal (time-series, not static snapshot) that the static data we'd been testing structurally cannot provide. We shipped real-time snapshot infrastructure the same week, specifically to start accumulating the kind of data that *can* answer this properly, going forward, instead of retroactively.
4. **We built a public, live accuracy page** — not a static case study frozen at the moment it looked best, but a number that updates as ORION makes more real predictions and they resolve, visible to anyone, without an account. See it here: **[orion-accuracy.html](../orion-accuracy.html)**.

## Why this is the actual pitch

Every institutional buyer of a research tool already discounts vendor-supplied accuracy claims — correctly, because most of them are unfalsifiable. What we're offering instead is the methodology itself: the exact hypotheses, the exact sample sizes, the exact honest result, published under our own name, before we needed it to close a deal. If our model's accuracy ever becomes a real differentiator, it will be because a fund checked our public track record and confirmed it themselves — not because we told them to trust us.

---

*ORION Biotech Intelligence is a research support tool. Nothing in this document is investment advice. Past backtest results and past prediction accuracy do not predict future performance.*