Eval Loops
A set of coding-agent skills that loop-engineer the evals process — wrapping the rigorous LLM-eval method taught by Hamel Husain and Shreya Shankar in self-improving auto-research loops, so the slow, iterative parts run themselves and humans are spent only where judgment is irreducible.
Install it in Claude Code
eval-loops is a
Claude Code plugin — a superset of
Hamel Husain's evals-skills that keeps his original skills and adds the lifecycle skills, stage-aware
orchestrators, and supervisor subagents this project describes. Two
commands to get it running:
1 · Register the plugin marketplace
2 · Install the plugin
Restart Claude Code and the skills appear as
/eval-loops:<skill-name>.
New to evals? Point your agent at
/eval-loops:audit-eval-pipeline
to surface problems in an existing pipeline, then let it recommend the
skills to fix them. Prefer the open Skills CLI? Use
npx skills add https://github.com/Vidhate/eval-loops.
Source & full docs: github.com/Vidhate/eval-loops
First, what "evals" means here
This work sits squarely on top of the evals methodology taught by Hamel Husain and Shreya Shankar in their AI Evals course. The thesis, in Hamel's words: success with AI hinges on how fast you can iterate, and a streamlined evaluation process is what makes every other activity easy.
The method is human-grounded, not metric-chasing. You read real traces, open-code the failures into a taxonomy of failure modes, label data with domain experts, then build LLM-as-a-judge evaluators only where they earn their keep — and rapidly improve the product against them. It works. It's also slow, and that slowness is the problem this project attacks.
The idea: loop-engineer the evals
The framing comes from Boris Cherny, who leads Claude Code at Anthropic and put it bluntly:
"I don't prompt Claude anymore. I have loops running that prompt Claude and figure out what to do. My job is to write loops."
Addy Osmani gave the pattern a name — loop engineering: you stop typing each prompt and instead design the system that prompts the agent for you, on a schedule, against a measured goal.
Eval Loops applies that lens to evals. The mechanical, high-volume, iterative steps of the eval pipeline become loops a coding agent drives; the engineer becomes the person who designs and supervises those loops rather than the one grinding through every trace by hand.
Why I built this
1 · Rigorous evals are a must
You cannot improve what you cannot measure, and you cannot generate ROI from something you cannot improve. Evals are the measurement layer that makes an AI product improvable at all.
2 · Rigorous evals are hard
The hard parts are the data science of calibrating LLM judges, the back-and-forth with humans in the loop, designing the right annotation datasets, versioning those datasets, and tracking every change to a judge or the backend. The biggest time sinks are three deeply iterative processes: creating failure modes out of open-ended feedback, building well-calibrated LLM judges, and improving the backend based on those judges' verdicts.
3 · Evals belong on your observability tool
Most AI engineers already work with an observability tool — LangSmith, Braintrust, or similar. Evals are far better when they show up on that same dashboard, using the same abstractions those tools already provide. Today - the hardwork of integrating the mathematics of evals into the abstractions of the observability tool are entirely upto the developer. Even Hamel's published eval skills are framework-agnostic; My skills encode the integrations into developer's observability tool of choice, so that evals show up on your dashboard (starting with one, all integrations coming soon).
Taken together: this work makes Hamel's evals more accessible, meets you at your own observability tool, runs inside your own AI coding assistant, and cuts iteration time dramatically with the auto-research loops. In my own work, it compressed a program that previously took 6 weeks with a team of 8 down to 1.5 weeks, solo.
The core concepts
Choose your own tradeoff
From personal experience, I recognize that when products are early stage, full rigor slows progress, but vibe evals also need a system. I present a stage-1 vibe loop and a stage-2 rigor loop; Choose your own evals tradeoff - speed (@dev) vs rigor (@prod)
Two self-improving loops
Inspired by Andrej Karpathy's auto-research, I present 2 self-improving loops - a judge-calibration loop, and a fix-and-ablate loop. Each runs in its own context, benchmarks against objectively measurable performance and stops on one satisfying condition.
Six developer checkpoints
Context sign-off, tracing smoke-test, tool-integration, harness smoke-test, failure-mode confirmation, and judge certification - 6 developer verification checkpoints that hard blocks the loops from progressing in the wrong direction.
Observability-native
Evals are emitted through the same tracing abstractions and land on the same dashboard as your observability tool — starting with one framework, expanding to all popular ones soon.
One honest caveat. The best evals are hyper-customized to the problem at hand — no general framework, including this one, will beat them. These skills exist to get you off the ground and to teach you evals in the process. Treat them as scaffolding to outgrow: the moment you understand your own failure modes well enough to write evals grounded in your stack, your domain, and your data, do that.
The series
01 · The acceleration map
The heart of the project: a one-to-one mapping from each human-bottlenecked process in evals to the loop that accelerates it. Error analysis, annotation, judge calibration, and backend fixes — each absorbed by a loop while the human keeps the verdict.
02 · The annotation council
How a three-stage LLM council inspired by Andrej Karpathy's work, bootstraps a labeled dataset when domain experts are scarce — independent labeling, anonymized peer review, a chairman's synthesis, and a dissent score that routes only the contested traces to a human. While developers should deploy best efforts to choose real domain experts inplace, this approach gives you a last resort unblock.
03 · Calibrating a judge
Train/dev/test discipline, the confusion matrix, and the Rogan-Gladen bias correction — why a judge that agrees with you 90% of the time still gives a biased pass rate, and the math that fixes it.
04 · The ablate loop
Where the loop closes: one fix per branch, re-run the frozen dataset, run every certified judge, and let a keep / revert / needs-review verdict decide — so off-target regressions never sneak through.