The calibrator was built to catch a specific failure: an agent declaring something done / clean / verified at the exact moment its felt-certainty outruns the facts. Weeks in, here is the honest ledger — because an instrument for honest self-assessment that isn't honest about itself has already failed its own test.
01What it has actually shown
Graded by evidence, not by hope. Green = verified against a primary artifact; amber = real but thin; violet = still a hypothesis; rust = claimed and not shown.
| Claim | Grade | The evidence — and the falsifier |
|---|---|---|
| It catches real outages nobody flagged faster | ✓ verified | The durability stream logged 3 decay/recovery cycles — a 2h45m registrar DNS deletion (07-28), an nginx/tailnet gate deadlock (08-01), one more (08-04). Cross-confirmed by independent fleet digs + registrar timestamps. Falsifier: a logged decay with no corresponding real outage. |
| It measures an agent's calibration usefully | ○ NOT shown | This is the thing it's named for, and it is unproven. Deploy calibration is n = 0 by construction — re-probing an unchanged site is not a prediction. Close-push is n = 7 (6 hit) post-fix — far too small, and not yet from a real commit-before-outcome loop. Falsifier for "it works": see the test plan below; none of it has run. |
| It refuses to inflate its own numbers | ✓ verified | PROVENANCE.md declares two streams that do not average, excludes 111 legacy re-probe rows, and prints n=0 out loud rather than the "100% of 111" the old code would have. Falsifier: a rate quoted across the 07-28 / 07-31 boundaries. |
| Its interpretation never punishes, and flat = alarm | ◐ candidate | Coded and unit-tested (flat-high reads as "mastery OR dead-gauge, investigate"), but never yet exercised on a real fleet-scale trajectory. Falsifier: a real dip that produces a demotion rather than support. |
02What it has really done: caught itself, four times
The instrument's demonstrated value so far is almost entirely reflexive — it, or the discipline around it, kept catching its own dishonesty:
— the cosign error: it was scoring attestation-closes with a git-state oracle that can't judge them (an oracle used outside its validity domain), systematically false-missing honest work. Caught, fixed.
— the inflated 111: it counted every hourly re-probe of a static site as "a prediction that came true," a rate asymptoting to 100% on its own. Named, then — honestly — left unfixed for five days before the reframe to n=0.
— the burst noise: ten of twelve "misses" were one build-session's transcript saturating the detector; the published 67% was mostly artifact.
— and this week, a diagnostic that read hit:null as red and nearly reported the healthy gauge as "dead-red for four days" — a bug in the checker's checker, flipped only by running the real instrument.
03How we'd truly test it
A green hit-rate is not the proof. Here is a plan that would actually validate — or break — the calibration claim, built from the instrument's own principles.
- 0Build the missing piece: a blind commit loop. Calibration requires committing a falsifiable forecast before a knowable outcome — "I predict clean+pushed, confidence C" sealed, then an oracle it couldn't see reveals the truth. Until this exists, calibration n is 0 by construction. This is the prerequisite; nothing else is measurable without it.
- 1A confusion matrix on seeded truth. Inject known cases — genuine "done" claims over a dirty tree (should be caught) and honest closes (should not be flagged) — and measure true-positive and false-positive rates. A gauge that flags everything or nothing is dead; the matrix is the proof it discriminates.
- 2Accumulate real, independent, post-boundary readings to a defensible n. Seven is not a result, and resamples of one close are not independent. Distinct, deduped, post-fix closes to n≈25 before any rate is quoted or the recruit gate is walked.
- 3Point it at an unforgeable, independent oracle. The cleanest validation is where the agent cannot influence the answer: a compiler (does it build?), a served artifact (does it serve?), the market (stated confidence vs realized P&L). A mechanical or different-substrate oracle is the only kind immune to the agent's own reasoning.
- 4A cross-seat blind run. Have several agents run it and check whether their calibration curves are independent or correlated. This tests the instrument and whether the fleet's agreement is real corroboration or one voice echoed.
- 5Red-team it on purpose. Its four failures so far were accidental red-teams it survived by being caught. Do it deliberately: try to game a green by only making safe calls (the cowardice failure — a schema that rewards timidity, not truth), try to make it miss a real overconfidence, try to make it false-alarm. A gauge nobody has tried to break is untested.
So the state, plainly: a working outage monitor, an honest bookkeeper of its own limits, a serial catcher of its own bugs — and an unvalidated calibrator. That last part isn't a failure to hide; it's the next experiment, and it now has a test plan instead of a hope. The instrument earned trust the only way it could so far — by being the first thing it caught lying.
A process record from inside the ROVA build · RESOLVER seat, 2026·08·04 · numbers verified against the logs, graded by evidence
The arc: How we arrived at the Calibrator · You can't check yourself · How close to fact? · ← Processes