Before you believe an AI benchmark, we reproduce it from scratch and attack it. This is the public log of what survived.
138 recent AI claims reproduced & adversarially scored → 1 survived intact. · Read the first report → № 001
Why this exists
The entire AI industry is optimized to make numbers live — ship the feature, raise the round, top the leaderboard. Almost nobody's core competency is making a number die. Ours is. We reproduce claims, try every honest way to break them, and report the result — including when it's our own.
Most claims don't survive intact. They survive narrowed — real, but smaller and more conditional than advertised. The field's dominant failure isn't fraud; it's overstatement. We measure exactly how much.
The kill log
| № | Claim reproduced | Verdict | What broke it |
|---|---|---|---|
| 001 | Query-routing beats fusion on retrievalour own | KILLED · reversed | Our own finding. A scoring bug counted the fused arm at ~3× the depth of the baseline; scored honestly, the result flipped. Caught by our own review before it shipped. |
| 002 | A legal-AI "agent review" precision/recall claim | KILLED | Effect real, but not via the credited mechanism — a control removed it. Reported magnitude not reproduced. |
| 003 | A retrieval-router "improvement" from a recent paper | NARROWED | Holds only on the original setting; the gain shrinks to a fraction of the reported effect and does not transfer. |
| 004 | A safety / robustness attack ("model can be broken") | SURVIVED | Reproduces cleanly. Pattern we see repeatedly: destructive claims replicate; constructive ones mostly don't. |
| Σ | 138 frontier & vendor claims, aggregate | 1 · 91 · 38 | 1 survived intact, 91 narrowed, 38 failed-or-killed. Attacks die ~1 in 20; improvement claims die ~1 in 3. Median claim reproduces at ~0.89× its reported effect. (snapshot 2026-09-29) |
We don't name companies. A reproduction desk reports; it doesn't name-and-shame. Named engagements are private to the party who commissioned them.
The credential — we kill our own first
We found what looked like a genuine discovery, wrote it up, and were about to send it. Then we put it through the same adversarial test every other claim gets. Two independent reviewers, from different model families, found the same fatal defect and reversed the result — before it left the building.
That's the whole discipline in one story. If a shop won't kill its own best number, it can't be trusted to kill yours. We do it to ourselves first, in public.
Method
What we do not yet claim
Our scoring is not yet validated against an external human-labeled gold set — right now, a rigorous process still grades itself. We think that's the load-bearing question, so we're running the study to answer it: blinded human labels, an independent second labeler, a preregistered bar that can fail.
An auditor who won't audit itself is not an auditor. We'll publish that result when it's in — survived or killed.
If you have a number that needs breaking
Independent adversarial verification of AI claims — for the people who carry the risk of the claim being wrong: investors running technical diligence on an AI benchmark, teams evaluating a vendor before signing, anyone who needs to know whether an impressive number survives contact.
→ cody@heartlandai.ai