The benchmark
Does 70% actually mean 70%?
A prediction is only useful if it’s honest about its own uncertainty. We run the full auditor over a held-out labeled set and measure two things independently: can it tell replicated from failed (discrimination), and do its probabilities mean what they say (calibration). The numbers below are computed live from the harness, not asserted.
Real papers, real outcomes
Sanity-check floor on 24 externally-labeled papers (6 harder / contested), full pipeline, live
The clearest signal: mean Litmus likelihood climbs monotonically with real reproducibility.
Every row is a live audit reproducible from its DOI. This is a sanity-check floor on clear-cut and harder/contested cases, not proof the engine is perfect on genuinely ambiguous papers; the AUC carries a wide confidence interval at this sample size and the honest next step is to widen the slice toward the full corpora. Retraction detection legitimately contributes on the retracted rows. Check mark threshold is 50%.
Calibration harness
Validating the calibration math at scale
The real slice above proves discrimination on real outcomes but is small. To validate the calibration machinery (isotonic fit, per-field curves, reliability, ECE) at statistical scale, we run a synthetic set whose per-field base rates are drawn from the published replication literature. It is a harness for the math, not a claim about real papers, and is clearly labeled as such.
Calibration
Reliability diagram on held-out papers
Raw model scores are over-confident (they sit off the diagonal). Fitting isotonic calibration on the labeled training split pulls them onto it, ECE drops from 0.144 to 0.093. Because isotonic is monotone, discrimination (AUC) is unchanged, calibration is a free win on top of it.
Ablation
Each component earns its place
Discrimination climbs as we add signals: deterministic checks alone, then retrieved literature, then adjudication, then adversarial verification. Each rung is calibrated on train and scored on the same held-out set, nothing is arbitrary.
Against the baselines
Discrimination (ROC-AUC)
no signal
calibrated but non-discriminating
text/metadata models (Yang, Youyou, Uzzi et al.); documented to degrade out-of-sample
this run
Per-field calibration
Base rates differ by field, so we calibrate by field
| Field | Base rate | AUC | ECE | n |
|---|---|---|---|---|
| cancer preclinical | 40% | 0.78 | 0.134 | 65 |
| social psychology | 39% | 0.85 | 0.070 | 65 |
| biomedical | 50% | 0.84 | 0.084 | 65 |
| economics | 61% | 0.85 | 0.134 | 55 |
A single global calibration would lie: psychology replicates at ~39%, economics at ~61%. One curve per field keeps the probabilities honest everywhere.
Two evals, kept separate on purpose. The real slice at the top runs the full production pipeline on real papers with externally-sourced outcomes (Retraction Watch, Registered Replication Reports, Nobel/Turing-recognized foundational work); it is the honest test of discrimination, and it is deliberately small. The synthetic harness here uses a labeled set of 500 papers (per-field base rates from the published replication literature), split 50/50; the calibrator is fit on train and every metric is computed on held-out test by the same code that powers the product. It validates the calibration math at scale, not real-paper performance. Next step: extend the real slice toward the full corpora (RP:CB, RP:P, DARPA SCORE) with the identical harness. The bar to beat: published ML replication predictors reach ~0.68 AUC and degrade out-of-sample.