Skip to content
Adi's Digital Garden

Project portfolio · deep dive

From clinical noise to trial-ready cohorts

ConcertAI held billions of unstructured patient documents — nurse notes, phone calls, lab reports, imaging, pharmacy and billing records — for 4 million oncology patients across four decades. Genome360 mined that text to extract genomic biomarker results and surface exactly which patients qualify for which clinical trials — the data pharma pays the most for.

The solution, in three parts

1 · Find A hierarchical classifier cascade surfaces the rare NGS reports from billions of documents.
2 · Extract NLP pipelines read the biomarker names and their mutation values out of each report.
3 · Verify A two-panel web app lets a nurse practitioner confirm every extraction against the source — clinical ground truth, human-confirmed.
Each part is its own deep-dive page — together they turned an un-queryable text dump into a structured, trial-grade genomic dataset.

The outcome

30K
NGS patients surfaced
~8
cancer types covered
~8,400
person-years of manual work avoided
#1
most valuable data product at ConcertAI

What automation bought — time, on a log scale

Human turnaround to build Genome360 by hand. Finding the reports is the order-of-magnitude win; automating extraction adds another. Together: from impossible to a few person-years.
Detail — how these are estimated

Per-unit assumptions, deliberately conservative: 15s to open and classify one document, 30s to read and transcribe one biomarker key-value pair (× 800 ≈ 6.7h per report), 5min to verify a pre-extracted report in the human-in-the-loop app. Over a corpus of ~4B documents yielding ~30K NGS documents:

  • No automation = screen 4B docs (15s) + extract 30K reports (800 × 30s) ≈ 16.9M hrs — screening dominates.
  • Step 1 only = classifier surfaces the reports; humans extract 30K (6.7h) ≈ 200K hrs.
  • Steps 1 + 2 = humans only verify 30K pre-filled reports (5min) ≈ 2.5K hrs.

Person-years at 2,000 work-hours/year. The point is the order of magnitude, not the decimal.

The context — ConcertAI

What it is

AI-driven SaaS & data products in US cancer healthcare.

Its customers

Pharma — Johnson & Johnson, Pfizer, BMS — enrolling patients into trials.

Its suppliers

Hospital networks across the US, supplying the raw patient-record dump.

My role

I owned the algorithms pipeline end-to-end — the cascade that finds NGS reports and the NLP that reads them — which made me the cross-team point of contact and pulled me into adjacent surfaces (a custom OCR pipeline, the validation-app backend).

End‑to‑end
algorithms pipeline owned
v1 → v8
shipped 0→1 and 1→10
Award
ownership beyond data science

Go deeper

Ordered as the story flows — from why it's hard to what it was worth.

01 · Problem & Context →

NGS & biomarkers, the 1:5,000 imbalance, why the old keyword search failed, and the spec.

02 · Finding the Needle →

The 3-stage hierarchical classifier cascade and AWS Batch scale-out over billions of docs.

03 · Extracting Biomarkers →

NLP pipelines that pull biomarker names and mutation values out of each report.

04 · Human-in-the-Loop →

The two-panel validation app and why clinical truth has to be human-confirmed.