Post 01
The acceleration map
One-to-one: every human-bottlenecked eval process, and the auto-research loop that accelerates it. In each case the loop absorbs the volume and the human keeps the verdict.
The Evals Process
Before the value of loop-engineering evals can land, it helps to understand how evals are actually run on an AI app. The best way to get that context is Hamel and Shreya's course on Maven — but in short, it's the nine-step process the cards below capture:
- Context — pin down what the app is meant to do and what "good" output looks like.
- Tracing — instrument the app so every input, intermediate step, and output is captured.
- Harness — run a batch of representative inputs through the app's backend to produce traces.
- Dataset — freeze those traces into a fixed, versioned set you can measure against.
- Annotation — a human reads each trace and labels it pass or fail.
- Error analysis — cluster the failures into a handful of recurring failure modes.
- Build judges — build an automated detector for each failure mode (a code check where possible, an LLM judge only when forced).
- Calibrate judges — tune each judge until it agrees with the human labels.
- Fix & ablate — change the backend one thing at a time, re-running to confirm each fix helps without regressing.
These steps aren't a menu you pick from — the order is an implicit dependency chain, where each step is gated by the one before it:
- You can't fix & ablate the app until you have automated judges to score it.
- You can't trust those judges until you've calibrated them against human labels.
- You can't calibrate a judge before you have labels.
- You can't label meaningfully before you know the failure modes worth catching.
- You can't find failure modes before traces exist to read.
Traditionally, this whole sequence is spread across cross-functional teams — surfacing failure modes can fall to a Product Manager, while calibrating judges is done by data analysts in collaboration with AI engineers. The highlighted steps below are the ones that used to eat the calendar; eval-loops now drives them for you. Four of them — annotation, error analysis, judge calibration, and fix & ablate — are full self-improving loops, and I present each through this skill.
Dashed accent = previously caused large time-sink; now eval-loops drives autonomously; devs verify and steer.
Solid = one-time implementation; previously no reusable framework covering all observability tools; now eval-loops covers specific observability tools and executes, devs verify.
First - Speed vs Rigor
The nine steps above describe the fully rigorous eval process — but from experience, how much of it you should actually run depends on where you are in building your idea. The real tension is speed versus rigor, and the right answer shifts as your conviction grows.
Early on, fast progress matters more than perfect features — you're trying to build conviction that the idea is worth pursuing at all. Here the right evals are the ones that give you a quick gut-check that you're moving in the right direction, even if they aren't airtight. Optimizing for rigor this early just slows you down before you even know the idea is worth the investment.
Once conviction is proven and the stakes — and resources — are high, the priority flips: your features need to be bulletproof. Now your evals have to measurably prove whether a feature works, and when it doesn't, point you at the exact vector to improve it against a metric. Getting it right matters more than getting it fast.
That trade-off is why the skill ships two
orchestrators on a maturity ladder. Which one runs is decided by a single
Stage field — because the right amount of
rigor depends on how much you yet believe in what you're building.
Stage 1 · Vibe
Solo dev, pre-conviction
- Signal in under an hour — file logging, ~10 inputs, in-chat review.
- One loop: generate → run → eyeball → fix → re-run.
- Skips dataset versioning, judge calibration, formal error analysis.
Stage 2 · Rigor
Team with experts, real stakes
- Trustworthy measurement — versioned data, calibrated judges, CI-ready.
- Four loops: triage · council · calibrate · ablate.
- Six human checkpoints that never auto-resolve.
The vibe loop is a single fast cycle — generate inputs, run them, eyeball the traces, fix, re-run — built to produce signal in under an hour and nothing more. The rigor loop is the full machine: versioned datasets, calibrated judges, and the four acceleration loops below, each gated by human checkpoints that never auto-resolve. The vibe orchestrator's real job is watching for graduation signals — three loops in a week, experts show up, the agent goes live — and handing off to rigor when the idea has earned it.
These two orchestrators are your entry points for running the process: you invoke one of them, and it manages everything beneath it — every sub-skill, every subagent, and every loop on this page. Nothing here is run by hand; it all hangs off the orchestrator you start with.
Crucially, an orchestrator is a long-lived supervisor that never reads a raw trace itself: it spawns a subagent per heavy step in that subagent's own context window and receives back only a summary. That's the move that makes nested loops viable — the volume work happens where it can't poison the supervisor.
The acceleration map
The heart of the project. Each card is a traditionally slow eval process, the loop that loop-engineers it, and the verdict that stays with you.
Annotation → LLM council
Reach for this only when you have to. Labels from real domain experts are always the better choice — use the council as a stopgap when experts aren't available, never as a replacement for them.
- The bottleneck
- Labels gate everything downstream, and they wait on scarce domain experts.
- The loop
- A three-stage council — independent labeling, anonymized peer review, chairman synthesis — produces a baseline label set with a dissent score per trace.
- Stays human
- Experts review only the high-dissent traces the council flags, not the whole set. Council labels are a stopgap, never ground truth for a judge you'll trust.
Error analysis → triage loop
- The bottleneck
- A person reads hundreds of failing traces and hand-clusters them into failure modes — days of open coding.
- The loop
- Reads 30–50 fail traces in isolation, bottom-up buckets into 5–10 modes, scores each by stakes, then splits them into two work queues. Returns a catalog, not the traces.
- Stays human
- The failure-mode confirmation checkpoint — you sign off the catalog before anything builds on it.
Judge calibration → calibrate loop
- The bottleneck
- Tuning an LLM judge against human labels by hand — tweak the prompt, re-score, eyeball, repeat.
- The loop
- Manages train/dev/test splits, runs calibrate → measure → refine → re-measure, spawns a refinement subagent per iteration, and measures the test set exactly once.
- Stays human
- The judge-certification checkpoint — you see the test TPR/TNR and the bias-corrected production rate before the judge is trusted.
Fixing the agent → ablate loop
- The bottleneck
- Change one thing, re-run the dataset, re-run every judge to catch off-target regressions, decide keep or revert — for every failure mode.
- The loop
- One fix per branch, re-runs the frozen dataset, runs all certified judges, computes deltas vs baseline, and issues a keep / revert / needs-review verdict.
- Stays human
- Any needs-review verdict and any guardrail regression pause the loop and surface to you.
The fork that lets the rest run itself
The triage loop's most important output isn't the failure-mode catalog — it's the spec-vs-gen split. Every failure mode is routed down one of two arms, and that routing is what makes the downstream loops able to operate unattended.
The requirement was never in the prompt, a tool is missing, or it's an engineering bug. The fix is one atomic, low-ambiguity change.
→ coding agent fixes itThe instruction is in the prompt but the model won't follow it consistently, or detecting the failure needs interpretation — tone, faithfulness, helpfulness.
→ build & calibrate a judgePrioritization across both queues is a single explicit score, so the loops know what to attack first without asking. Frequency-only prioritization is the common mistake; stakes and stage keep a rare-but-catastrophic mode from sinking to the bottom.
Self-termination: the calibrate loop
Loop-engineering is only safe if the loops stop on their own for the right reasons. The judge calibrator iterates on the dev split and refuses to touch the test split until the very end.
The loops won't quietly overstep. Looking at the test set during iteration, auto-swapping the judge's model when stalled, or treating council labels as ground truth are all explicitly forbidden — the loop surfaces the decision instead of taking it. Acceleration is scoped to the mechanical work; the judgment calls escalate.
Where the humans stay
The Stage-2 orchestrator holds six checkpoints that are hard blocks — it cannot chain past them, no matter how confident the loops are.
The point of the loops isn't to remove the human. It's to make sure the human's attention lands on these six decisions instead of being burned on the thousand trace-reads and prompt-tweaks between them.
The throughline
Two ideas carry the whole design. Context isolation is what makes nested loops viable at all — a supervisor that read every trace would collapse under its own context, so reading happens in workers that return summaries. And measurement gates every loop — nothing is kept, certified, or merged without a number it had to clear. Take either away and "loop-engineering the evals" becomes an agent confidently making the eval worse. Keep both, and the slow middle of the eval pipeline starts running itself.