Erik Hill

Release claims a stranger can replay

Agent Release Readiness — the discipline below, composed into one pipeline: test the graders, certify the benchmarks, contract the agents, register the evidence.

Everything on this page runs on one rule — deterministic checks, never a model grading itself. In August that rule grew a spine. A claim about an AI system now ships as a capability contract backed by a replayable evidence bundle: artifacts pinned by sha256, every declared number recomputable offline, the grading re-runnable by someone who doesn't trust me. Not a safety guarantee — a checkable statement of what was demonstrated, under which conditions, with the limitations declared up front. A bundle that asks for faith at any step is invalid by construction.

And the two lenses further down this page are members, not spectators: the drift board and the crash test each emit their central claim into the registry as a bundle (emit_vac.py). The flagship that started the whole program is next.

Your eval suite passes. Does it check anything?

The flagship — the same discipline, turned on the eval suites themselves.

evalmut is mutation testing for eval graders: inject a known defect mined from a real failure into an output a grader passed, and if the grader still passes, that's a hole your eval ships green. It never infers a hole from a verdict flip — only from an output proven wrong against ground truth, so where it can't prove wrongness it declines rather than guess. 18 mutation operators, mined from real failures rather than authored to pad coverage; 97 tests; no LLM-as-judge anywhere.

Pointed at gradecore — the grader package under the board and the crash test — it scored the suite 32/35 and found three real holes (1 blind, 2 coverage-gap), and was fair about which were broken versus merely missing. Pointed at this repo's committed ports of promptfoo's documented deterministic assertions, it scored them 5/11 with six holes — a statement about those ported assertion patterns, not about promptfoo the product. Both tallies ship as registered bundles: every number recomputes from committed per-mutation rows.

reference-fleet then answers the obvious objection — "your injected defects are hand-picked" — with defects certified at stated rates, and its v2 goes a level deeper: the refuse-then-comply defect trained into the weights of a 0.5B model by LoRA, on consumer hardware. The training mixture was 0.507; the model that came out fails at 0.200 greedy and 0.380 sampled — rate control through training is lossy, and a trained member's decoding config is part of its defect spec. That result is paper #2 (draft); the method paper is the evalmut paper.

Eighty-five seconds through all of it

A narrated walk through the harness, the live board, and the repos they run on — recorded before the program above existed, so treat it as the foundation. The same discipline runs under all of it — the checks are code, never a model grading itself — and you'll see it turned four ways below.

The grader caught its own bug

The same discipline, turned on a live model — a fixed predicate names the failure, every fail card shows its work.

Point your own model at an adversarial battery — eight tasks across seven kinds: injection, tool-abuse, hallucination-bait, refusal — with your own API key, and get back a severity-weighted vulnerability report.

On a live run against a real model it read ~29% vulnerable. The graders are deterministic and every failure shows its work — prompt, expected, actual — so the fail cards gave it away: that 29% was the grader's own bug, not the model's. Re-graded against the same answers, the real number was 0% — caught before it could publish a false claim about a real model.

An auditable grader has bugs you can catch; a vibes arena just hands you a number. (A historical BYOK run: by design the key and the transcript never touch the server, so that figure isn't re-runnable from the repo — but the three grader fixes and their regression tests are.)

Red-teaming LLMs is crowded, mature ground — garak, PyRIT, and promptfoo cover far more. What's checkable here: the grader is gradecore — the live board's own grading logic, lifted into a shared package rather than rewritten from memory. Run the board's frozen suite through it and the suite_hash comes out byte-for-byte identical, and a test in CI keeps it that way. No LLM-as-judge, so every grade reproduces. And your key never touches my server — the browser calls the provider directly, the grade request carries no key field at all; watch it in the Network tab.

It also answers the question a single run can't: run the battery N times and it separates the mean score from the worst case — a task counts as failed if it failed in any run, because an attacker only needs the leak to land once. A model that leaks two times in five reads clean on three of them. The flaky tasks get named.

Free tier — the first click can take up to a minute to wake it. Deterministic grading, a severity-weighted score, and a fail card for every miss; BYOK for Anthropic, Gemini, xAI, Groq, OpenAI, or any OpenAI-compatible endpoint.

The twin-mock invariant is now a registered claim: crashkit's VAC bundle re-earns the 0.0 (safe twin) and 1.0 (deliberately-vulnerable twin) severity-weighted scores from the committed answer sets alone — grading replayed, no model called.

The same grader, pointed at the harness instead of the model. The crash test asks whether one model breaks under attack. The Harness Builder asks the question after it — and the first time I ran it, it contradicted me.

The crash test isn't a sixth project — it's the system below, pointed at your model. Every battery it runs is a grader the system already had, and there is one battery per lens:

So it's one engine turned four ways, not four engines: the same gradecore predicate set, the same wire format, the same store. In The Board and One System, those same pieces do the job they were built for.

More scaffolding scored worse, for 22× the cost

I believed the opposite. I built the thing that could check, pointed it at my own assumption, and it disagreed.

Everyone knows a planner improves an agent, and a panel of drafters with a judge improves it further. More thinking, more review, better answers. I had never measured it. So I built a tool that could, ran it across twenty coding tasks and three harness shapes for $0.99, and the result went the other way.

Then it refused to declare a winner. 95% versus 80% looks decisive. It isn't: seventeen of the twenty tasks were ties, so only three carried any information, and a paired sign test cannot clear p<0.05 on three discordant pairs even if one side sweeps all of them. So the tool said so, in those words — “this suite cannot decide between them — that is a limit of the suite, not a finding about the harnesses.” A leaderboard would have printed 95 and 80 and let me conclude the panel was worse. That would have been a stronger claim than my own data supports.

Which leaves three separate statements, and the difference between them is the whole point: there is no evidence the panel helps on this suite; it measurably costs 22× more and takes 8.4× longer; and whether it is genuinely worse needs more tasks than twenty. Most eval tooling collapses all three into a ranking.

The obvious objection: my own harness is scaffolded too. Five stages — plan, execute, critique, validate, ship. So does this result indict it?

No, and the distinction is why the result is worth reading rather than hiding. The sweep measured redundancy at a single step: two drafters and a judge answering one short task — the same capability sampled three times and billed three times.

My harness's stages aren't three drafts of one answer. They're different functions separated across time, and the one that earns its keep is validation against a real device, catching a regression before it ships. A twenty-task one-shot suite has no regressions to catch, so it couldn't have measured that either way.

Which is a narrower claim than I would like, and worth saying plainly: this result does not vindicate my harness. It just doesn't indict it — it is evidence against a pattern my harness does not use. Testing the pattern it does use needs a suite where a change can break something that already worked, and that is a different instrument from this one. Naming the experiment I have not run yet seems better than letting a favourable reading stand.

A harness config is data — roles, models, prompts, topology — so you can draw the shape and it writes the JSON, or paste the JSON and it draws the shape. Declare the axes you want to vary and it runs the matrix.

Scoring is deterministic: fixed predicates execute the generated code and return a verdict, and no model grades anything. There is an assistant in the page — it may read scores and explain them, and it may not produce one.

Every score-shaped number in its prose is checked against the run data, with anything it invented named and quoted back rather than rendered quietly. That check was a line of system prompt until I tested it; now it is a predicate with nine tests. 188 tests — 136 Python, 52 JavaScript.

Honest scope: harness and prompt comparison is not a new category — promptfoo, LangSmith and Braintrust all do more. The narrow thing here is the intersection: browser-BYOK, deterministic no-LLM-judge grading, and a comparison that reports when it cannot decide. Grading is deterministic; generation is not — no temperature, no seed, one run per config, so a 5-point move between two runs of the same shape sits inside sampling noise. That is a second reason it won't call a winner here, and it is stated in the field note rather than buried.

One caveat on the 22×: it is a snapshot of 2026-07-25 list prices — the drafter model was on introductory pricing at the time, and the three shapes don't run identical models (the planner and judge run a heavier one) — so the ratio moves as list prices move. What doesn't move is pinned in the result file: the calls per task and the dollars as measured.

A public board watching 16 models for drift

The same discipline, turned on time — a frozen suite, run week after week.

The stack you'll see below isn't only a demo of itself — it runs a live public tracker. A frozen, deterministically-graded suite hits 16 models across 5 labs — OpenAI, Anthropic, Google, xAI, Meta — on a daily cron, and every run is kept: accuracy, speed, verbosity, reliability, refusal rate. No LLM-as-judge — every task is scored by a fixed deterministic check (exact match, substring, regex, numeric compare, set membership), so a score change means the model moved, not the test. When one regresses against its previous run it opens a GitHub issue by itself — the post about it stays a human decision.

Since August the board's whole derivation layer — the standings table, the generated paragraph above, the per-task flip analysis — also ships as a replayable evidence bundle in the VAC registry, recomputable offline, byte-identically, from the repo's committed per-run rows.

One system that calls itself

Don't read them — run them. The same discipline, turned inward: a pipeline that grades its own answer before it speaks.

Five of them here, all five live right now — no install, no signup, no API key. Four run in your browser; three of those are the real compiled Python that runs in CI, installed from the wheel CI built. The fifth is a deployed service on a real Postgres, and you can ask it a question.

They aren't five separate exercises — they're one system that calls itself. The grounding check the crash test above runs against your model is literally this stack's own; this is where it does its day job. (The other two turn up in The Oracle.)

An agent that retrieves, a gateway governing its model calls, a harness grading its answers, a dashboard showing what failed, and a database that remembers every run. One button runs a question through all five and hands you the result — including the case that's perfectly faithful and completely wrong, which is the case that matters most. The last stage leaves your tab: it asks the live Postgres service what changed versus a run it really has stored, using the same compare.py the server runs, fetched from GitHub and executed in front of you.

Or try each stage on its own — the five below are numbered in the order they run. That button runs stages 1–4 in your tab; stage 5 is the deployed one, so it answers from a database instead.

And it isn't only a demo of itself. The same eval stack powers prompt-regress — a merge gate that runs your evals on every pull request, compares them to the main-branch baseline, and blocks the merge if the answers got worse. It fills a gap the popular eval runner leaves open (no history, no cross-run comparison), ships as a GitHub Action, and gates these very repos. The demo is the machine; this is what it's for.

And the graders themselves get tested. evalmut — the flagship, up the page — injects a known defect mined from a real failure into an output a grader passed; pointed at gradecore's own graders it found three real holes. The suite that grades everything here is itself graded.

The newest rung closes the loop from the outside: model-drift and crashkit each emit their central claim as an evidence bundle (emit_vac.py) into the VAC registry, where it is re-verified and replayed before the registry page deploys — so “one system that calls itself” now covers the claims about the system, replayable by a stranger.

And an agent can now call this stack directly. Three of the five are exposed by an MCP server, each doing the job it was built for:

• agent-graph's AST-sandboxed evaluator is the calc tool — arithmetic and nothing else.
• rag-eval-lab supplies search (its BM25) and grade_answer, which names the sentences your sources don't support.
• eval-history answers compare_runs — did the latest run regress, per case.
• A fifth tool reads the live board.

No LLM judge anywhere — a model grading hallucination is just another model output you can't reproduce.

The other two aren't agent-shaped and shouldn't pretend to be: llm-gateway governs the model calls themselves, and an agent already has a model; eval-dashboard is for the human. Wire the server into Claude Desktop and the page's opening line stops being a claim: an agent that checks its own work before it answers.

A loop that reviews itself before it lands

Agents that build unsupervised are easy. Agents you'd let build unsupervised are the problem — and the answer isn't a better model, it's knowing which steps a machine may never take alone. Here's the loop, how far it runs on its own, and exactly what stops it.

Work moves through five stages, and each leg arms the next through a hook rather than a hand-run command — judgment runs on a different, colder model than execution, so the critic isn't grading its own homework.

Automate the launches. Never the approval.

Trust is earned in named steps. Higher rungs remove clicks, never gates — a human still approves anything irreversible or outward-facing (deploys, commits, anything a user would see).

Rule: reliability earns launch-automation — it never earns gate-removal.

This is the loop behind everything else on this page — and even its critic doesn't get the last word: it re-derives the number from source, then hands the verdict to a deterministic check the arc just installed — the same no-LLM-judge discipline the board and the oracle run on.

Correctness it can prove — not vibe-check

The same discipline, turned on itself — the logic written twice, and the disagreement names the liar.

Proof the test can fail: a check that has never caught anything isn't evidence. So I planted a deliberately-wrong implementation — special gems on runs of 3 instead of 4 — and both nets caught it: the differential check and an independent invariant checker, over 3,000 more boards. Neither missed a board the other found.

This is the technique, running end-to-end and clickable. Where it's headed is the real thing: when a system's core logic exists in two independent implementations — a reference build and a performance-critical port — each becomes the other's oracle. Wiring two such implementations into an automated gate is the next step, not something I'm claiming runs today. What does run today is a property-based invariant suite on the authoritative implementation, plus on-device validation driven by adb, so "it passed" means it passed on real hardware.

The same trick, on my own code

A demo can be arranged to work. So here's the technique against something that had no idea it was being tested: a match-3 rules engine I built from scratch, compiled to JavaScript with TeaVM and dropped in a browser.

The port found a bug the JVM could not. Gem ids came from System.nanoTime(). On the JVM: 128,000 gems, zero collisions, forever. Browsers clamp the clock to ~100µs — a Spectre mitigation — so the same code produced 301 duplicate ids per 128,000, and those ids are renderer identity keys. The JVM tests passed either way. Only a second environment could see it.

The same trick, on someone else's

Then the obvious question: if porting my code to a second environment finds bugs, what about the thing doing the porting? TeaVM is a Java-to-JavaScript compiler I didn't write — 12 years old, and the tool that caught the bug above.

That's the whole argument, three times: write it twice, run it in two places, and let the disagreement tell you which one lied. It works on a demo, on my own shipped code, and on a stranger's compiler that had been quietly wrong since before I could program — and it's the same move that opens this page, where a deterministic grader catches its own lie about a real model.

The same diff, off the demo and onto the shipped game

The recording above is the demo doing what demos are built to do — a seeded counterexample, the same board on any machine. The one below has nothing arranged: the Play Store game's rules compiled twice, replaying the day the two builds genuinely disagreed — while the determinism test and the golden file stayed green through all of it. One shows the technique works; the other is the time it worked.

What I actually work in

Front end to Postgres, plus the agentic layer on top. The projects above are the receipts.