Adi's Digital Garden

Deep dive · 04

Human-in-the-loop — the validation app

Part 3 of the solution. A split-pane web app where a domain expert checks the model's extraction against the original report — producing both the ground truth that retrains the models and the deliverable shipped to pharma. The interesting part wasn't the UI; it was what building it for 300 annotators taught me about data quality.

The split-pane app

Left: the report exactly as found. Right: the model's pairs — every value chosen from a controlled dropdown, never typed free-hand.

How a task flowed

Admin assigns An administrator hands out validation tasks, batched by cancer type.
Expert validates The domain expert creates missing pairs and corrects wrong ones against the source.
Export The finalized biomarker–value sheet is exported to persistent storage.
That validated sheet then did double duty:

Ground truth

Fed back to retrain and improve the Step 2 NER & RE models.

The deliverable

Shipped to pharma partners as the trial-grade genomic dataset.

~300
domain experts at peak, per cancer type
2 uses
model ground truth + pharma deliverable

Three lessons from building it for 300 people

The platform was straightforward; running it as a data-quality system was where the real engineering lived.

1 · Never ship an open text box

Annotator edits a biomarker or mutation value
Matches the fixed index
Accepted — submit & export allowed.
Off-index value
Flag raised · submit and export blocked until an administrator approves.
Every field — even editable ones — drew from a controlled index of valid values. No free text anywhere on the surface.
Detail — why free text is unfixable downstream

An open text box gets misused almost immediately: values come in un-normalized, non-standardized, and carrying errors that no downstream step can reliably repair. By constraining inputs to an index and hard-blocking off-index submissions, bad data never entered the pipeline in the first place. I stayed in close contact with the administrators to watch the flag rate and resolve new ones the moment they appeared.

2 · Match the document's own order

Raw model output order
TP53 → no variant
KRAS → G12C
BRAF → V600E
EGFR → exon 19 deletion
— annotator hunts up and down the page —
Sorted by NER position
BRAF → V600E
EGFR → exon 19 deletion
TP53 → no variant
KRAS → G12C
— reads in lockstep with the report —
One sort, by the NER model's original character positions, so the right-hand list tracks the left-hand report top-to-bottom — a trivial change that lifted annotator productivity substantially.

3 · Be reachable in the moment

Problem hit An annotator runs into something the tool handles wrong.
Memory fades fast Wait a day and the exact conditions that triggered it are gone.
Catch it live Being immediately approachable meant capturing the precise repro while it was fresh.
The most critical lesson: the only reliable moment to understand a problem is the moment it happens — so I made myself available to the annotators in real time.

← Back to extracting biomarkers