Deep dive · 02

Evaluations — how we knew it was safe to ship

In a clinical AI product, evals are the engineering moat. Knowledge quality and conversational quality are both downstream of how rigorously you can measure them — so this is the part of PHP I'd carry into any AI system I build next.

Helping a patient is two capabilities, not one

Right knowledge correct, current, relevant clinical fact

Right delivery empathetic, paced, usable by a human in distress

Actually helps if either factor is 0, the product is 0

So we ran two independent eval tracks — Knowledge & RAG and Conversational Behavior — each with its own taxonomy, annotation view, and judges.

Detail — why the split is a design decision, not a convenience

Collapsing both into one rubric produces noisy, un-actionable scores: a low rating could mean a hallucinated dosage or merely a verbose tone. Separating them makes every failure attributable to a subsystem we could actually fix. Perfect knowledge delivered coldly fails; warm delivery of a wrong fact is dangerous.

The failure taxonomy was discovered, not assumed

Open-ended Doctors dump unstructured feedback on every trace — nothing leading.

Cluster We compare notes across doctors & conditions, find recurring themes.

Bucket Recurring themes become named failure modes worth fixing.

Feed forward Top buckets become structured fields in the next round.

A rubric written up front encodes our assumptions and biases the reviewer — so we let the taxonomy emerge from the doctors.

Detail — the failure modes we converged on

Knowledge

Incorrect fact ingested
Irrelevant fact retrieved for the query
Incorrect query generated for a turn

Behavior

Didn't ask enough clarifying questions
Too verbose — should be brief & empathetic, with progressive disclosure once the patient signals value

The annotation tooling I built

Admin & governance · real patient PHI

Microfrontend UI

a new frontend per annotation task → an extensible eval platform, not a script

Task-specific ETL

reshapes each trace per task: Knowledge surfaces RAG tool-calls, Behavior hides them

Data layer

tapped at the core assistant — the source of truth for traces, no parallel silo

Three layers wrapped in governance. Off-the-shelf eval tools couldn't give doctors a bespoke per-task view over sensitive patient traces — so I built it.

Detail — same trace, two task-specific renderings

Knowledge view

patient › I missed my metformin this morning, what do I do?

tool › rag.query("missed dose metformin")

tool › retrieved: [fact#1182 dosing-window], [fact#0934 double-dose-risk]

ai › If it's been under ~2 hours…

Behavior view

patient › I missed my metformin this morning, what do I do?

— internal tool calls hidden —

ai › If it's been under ~2 hours…

The Knowledge task always captured relevance + accuracy fields on top of open-ended feedback. The Behavior task annotated turn-level on the full conversation, with the top failing buckets added as fields in later rounds.

What deserves a judge

A failure surfaces in review

Specification failure

Fixable to extinction → patch and move on. A standing judge would be wasted infrastructure.

Generalization failure

Recurs across inputs we can't enumerate → build an LLM-judge, because only continuous measurement catches regressions.

Eval infrastructure is a cost with an ROI threshold, not a reflex — we built judges only where measurement had lasting value.

The loop — slow human round, fast judge net

Monthly human round Real patients use the app; doctors annotate. Re-derives the taxonomy, catches new generalization failures.

Codify Generalization failures become LLM-judges, calibrated against doctor labels.

Between-rounds net Those judges run on every material backend change I shipped.

Prune Spec failures fixed to extinction drop out, keeping the judge set lean.

Monthly cadence was deliberate — we insisted on real chronic-condition patients across varied clinical stages, never simulated data.

Detail — why monthly, and who triggered it

Sourcing real patients with these conditions across different clinical stages is slow, but it's the only honest signal for a clinical product — simulated users would have taught the judges the wrong things. I triggered rounds when the agent backend had changed materially enough to warrant fresh feedback.

Calibrating — and governing disagreement

Multiple doctors Independently annotate the same traces.

Cohen's κ Quantify inter-annotator agreement.

Conflict calls Resolve divergence in discussion.

Clinical lead A head of clinical knowledge makes the benevolent-dictator tiebreak.

Subjective clinical judgment needs a governance model, not just a higher agreement score.

(Exact κ figures aren't on hand here — the point is the instrument and the disagreement governance, both of which were in place.)

Coverage & cost

chronic conditions

20 → 10–15

users/condition (start → peak)

~100–300

traces evaluated

4–5

chats/user, varied stages

Operational
(doctors + patients)

dominant cost

LLM-judge runtime

Human clinical expertise was the bottleneck, not compute — so judge cost wasn't separately tracked; it came from a bulk eval budget. The right bottleneck for this domain.

Detail — adversarial & red-team coverage

A separate adversarial / red-team set ran distinct from the core use-case traces, probing for off-scope advice, jailbreaks, and unsafe completions the naturalistic traces wouldn't surface.

The 0th round that reframed the product

Going in, the product owner and I had a confident definition of "good": deliver the most clinically accurate, important information as fast as possible. The very first round of evals demolished it. Almost none of the doctors' feedback was about facts or accuracy — it was all about behavior.

"Even though the AI understood the user's clinical standing and gave a clinically appropriate response, it should have asked more questions to get to know them — so the user feels heard about their symptoms."

"Don't reveal all the clinical help at once, even though it's accurate. Reveal it progressively, at the user's pace — what they're going through is big and needs empathy, not just information."

That feedback forced us to reassess what we were actually building. In conversations with the doctors, a striking split emerged:

~20% having the right knowledge

~80% the right behavior & character, so patients felt helped

Knowledge was the smaller half of the problem. This is the realization that split the eval program into two tracks — Knowledge and Behavior — in the first place.

The fix wasn't a code change — it was a definition change. The eval didn't just catch a bug; it caught a wrong premise. "Fast and accurate" became "accurate, then delivered with empathy and pacing," and every downstream rubric, judge, and annotation view followed from that.