Post 04
The ablate loop
Where the loop closes. Every fix gets its own branch, re-runs the same frozen dataset through every certified judge, and lives or dies by a measured verdict — so the change that fixes one failure mode can't quietly break another.
Off-target regressions are the real enemy
The most common reason a change fails in production isn't that it didn't fix its target — it's that it silently regressed something else. A prompt tweak that fixes tone breaks a constraint; a retrieval fix that helps faithfulness drops a guardrail. The ablate loop is built around that fear: it measures the whole judge suite after every single change, against an unchanging baseline.
First, an anchor
Before any fix, the loop runs the harness on the dataset's pinned
input_version and
runs all certified judges to record a baseline: the corrected pass rate and
confidence interval per failure mode. Inputs are pinned to a
static version on purpose —
dynamic inputs add variance that would drown out the signal of a single fix.
Every later experiment is a delta against this anchor.
One fix, one branch, one measurement
The backlog is the spec and gen queues from error analysis, ordered by
priority. Each item runs the same cycle on its own branch
(evals-fix/<fm_id>/attempt-<n>).
No stacking — stacking two fixes on one branch defeats the entire point of
ablation.
Spec fixes go to a coding subagent as a concrete edit, verified to touch only the expected files. Gen fixes — prompt restructuring, sub-agent decomposition, a model swap — propose first and pause for your review before applying. Running all certified judges, not just the targeted one, is the non-negotiable step.
The verdict rule
The deltas decide the fate of the branch automatically — except where they're ambiguous, which is the one case that comes back to you.
keep
Target improved beyond its CI, nothing else regressed past its CI, and no guardrail moved at all.
Merge to main. The new state becomes the baseline for the next fix.
revert
A guardrail regressed, more than one off-target mode regressed, or the target didn't actually improve.
Leave the branch unmerged (never deleted) and try a different fix next attempt.
needs_review
Anything ambiguous in between — a mixed result the rule can't cleanly call.
Pause the loop and hand the deltas to you. The supervisor does not self-resolve.
Guardrails are non-negotiable. A fix that improves its target while regressing a guardrail is a revert, full stop — there's no pass rate that buys back a broken guardrail.
When the loop stops
The supervisor holds only the backlog, the current item, the baseline, and the last few experiment summaries — trace bodies are dropped after judging. It halts on the first of:
Branches are never deleted, even when reverted — they're the experiment trail. Every attempt, kept or not, is a recorded result you can go back to.
Where the closed loop closes
Error analysis found the failure modes. The council labeled the data. The calibrate loop turned those labels into judges you can trust. Here, those judges become the measurement that lets a coding agent change the product and know whether it helped — one branch at a time, against a frozen bed, with guardrails that can't be traded away. That's the full loop: find, measure, fix, re-measure, keep what's better. The human stays for the verdicts that matter; the loop runs the rest.