Post 02
The annotation council
Labels gate everything downstream — and they wait on scarce domain experts. The council loop bootstraps a labeled dataset with a dissent score per trace, so a human only has to look at the traces the models actually disagree on.
The bottleneck it removes
You cannot calibrate a judge, run error analysis, or measure a fix without
labels. But a single LLM labeling a dataset is just one opinion, and asking
a domain expert to label hundreds of traces by hand is exactly the wait that
stalls a program. The council — modeled on Andrej Karpathy's
llm-council
pattern — gets you a defensible baseline fast: multiple models as
anonymized peer reviewers, a chairman to synthesize, and a number that tells
you where the humans are still needed.
Three stages
The council runs trace-by-trace through three passes. Each pass is a different prompt to the same set of models.
Stage 1 · Independent labeling
Every council model labels every trace on its own, in parallel — a binary Pass/Fail per failure mode plus a one-sentence evidence note citing specific trace content. No model sees another's answer yet.
Stage 2 · Anonymized peer review
Each model is shown the others' labels as Reviewer A, B, C — identities hidden — and asked to rank them by evidence quality and revise its own verdict if the others are better grounded. The peer-review step is what makes the labels worth more than a single model call.
Stage 3 · Chairman synthesis
A chairman model receives every label, every peer review, and the de-anonymized identity map, then produces the final verdict, an evidence note, and a dissent score per trace.
The dissent score is the product
The chairman emits a dissent score from 0.0 (unanimous) to 1.0 (an even Pass/Fail split), weighted by how highly the reviewers ranked each other. It's not a footnote — it's how the loop decides what to hand back to a person.
Low dissent
≈ 0.0The council agreed. The label is trustworthy enough to use as a baseline without a human looking.
High dissent
> 0.5The council split. This trace is flagged for human-review priority — it's exactly where an expert's time is worth spending.
Council labels are a stopgap, never ground truth. If you'll later calibrate a judge, that judge inherits the council's biases — so the labels it certifies against must eventually be confirmed by a domain expert. The council buys you speed and a triage signal; it does not buy you the right to skip human review where it matters.
Count the calls before you run
Three passes across several models adds up. The loop prints a cost estimate and asks you to confirm — and suggests a 10-trace pilot before the full run, because prompt sizes drive the real bill.
If the chairman call fails or returns malformed output, the loop falls back to a peer-ranking-weighted majority vote — and a tie breaks to Fail, the conservative default. The fallback is logged explicitly so you always know which labels were synthesized and which were voted.
What lands in the dataset
Only the synthesized result is written back into the dataset manifest as a new version — the per-stage transcripts stay on disk. Each label carries its provenance and its dissent, so downstream skills can treat a council label differently from a human one.