Deep dive · 02

Finding the needle — a classification cascade

Part 1 of the solution. Pulling NGS reports out of a 1-in-5,000 haystack isn't one classifier — it's a hierarchy of three, each preserving recall on NGS while progressively starving the imbalance, plus a zeroth step just to turn raw blobs into text.

Why one classifier can't do it

The accuracy trap at 0.02% prevalence. With one NGS report per 5,000 documents, a model that simply always predicts "not NGS" scores 99.98% accuracy and is 100% useless. The entire problem is recall on a vanishingly rare class — so I attacked the imbalance itself first, in stages, before asking any model to make the precise call.

The design principle for the whole cascade: preserve NGS recall at every step, spend each step removing true negatives, so the final precision-focused model sees a tractable ratio instead of a needle in a haystack.

Step 0 — get text out of anything

Raw file blobs — no extensions, no reliable filenames

Scanned images

Literal scans of patient notes → Tesseract OCR → text.

Text PDFs

Well-behaved documents → programmatic text extraction.

A file-format classifier sits in front, reading each file's binary header — because the dump arrived as nameless, extension-less blobs, and you can't route a document you can't identify.

Detail — why the format classifier was necessary

The raw blobs had no extensions and names that gave nothing away, so the only reliable signal was the file's actual binary signature. Identifying the format first determined the extraction path — OCR for images vs. direct parsing for text PDFs — and everything downstream depended on getting clean text out of this step.

The cascade — three stages, one funnel

Text-extracted corpus ~4B documents, all formats normalized to text

1 NGS : 5,000 docs

Step 1 · keyword true-negative filter — recall 0.99, drops ~90% of docs

Survivors of Step 1

1 NGS : 500 docs

Step 2 · XGBoost 11-class — keeps NGS recall 0.95, drops other classes

Survivors of Step 2

1 NGS : 5 docs

Step 3 · XGBoost binary — NGS vs non-NGS, recall 0.92

NGS reports

precision 0.85

Each stage enriches NGS concentration roughly 10× while holding recall — turning a 0.02% base rate into a set where the final, precision-focused model can do its job.

Step 1 — eliminate true negatives with doctors' keywords

Domain insight Doctors who read these documents daily named the keywords that mark true positives and true negatives.

Rule-based scorer Inclusion/exclusion on those keyword sets — cheap to run over the full ~4B corpus.

Drop ~90% Recall on NGS held at 0.99; prevalence rises from 1:5,000 to 1:500.

Optimized purely for recall: the cost of a wrong drop here is an NGS report lost forever, so the bar was to keep essentially all of them.

0.99

recall on NGS

~90%

of documents eliminated

1:5,000 → 1:500

NGS prevalence after Step 1

Step 2 — an 11-class model, and how I found "11"

Experts said 15 Domain experts proposed 15 document types — doctor notes, imaging, lab tests, NGS, irrelevant docs, and more.

K-means on TF-IDF Unsupervised clustering of the extracted text put the natural sweet spot at 11 clusters.

Merge 4 weak ones Four of the 15 had weak decision boundaries; absorbing them lifted precision and recall across all classes.

XGBoost · 11 classes NGS recall preserved at 0.95; everything not NGS-adjacent is dropped.

Without the K-means check I'd have trained on the experts' 15 — the clustering is what revealed that 11 was the cleaner, better-separated target.

Detail — still a recall play, not yet precision

The goal at Step 2 was the same as Step 1: keep removing true negatives without losing NGS. Classifying into document types let me drop entire non-NGS categories at once while watching a single number — NGS recall — stay high (0.95). NGS sits inside a broader molecular-pathology super-class, so the survivors are everything NGS-likely.

Step 3 — the precise call, NGS vs non-NGS

Top-K TF-IDF pruned to the most discriminative features by SHAP value on the train set

Nominal features document length, presence of very specific keywords, etc.

Binary XGBoost precision 0.85 · recall 0.92

Only here does the objective flip to precision — and only because the first two stages handed it a ~1:5 ratio instead of 1:5,000.

Detail — why SHAP-pruned features

The raw TF-IDF space was far too wide to feed in whole. Ranking features by SHAP value against the training set kept only those that actually moved the NGS-vs-non-NGS decision, then I combined that top-K set with a handful of nominal signals (length, specific keyword presence) that TF-IDF alone doesn't capture well.

0.85

final precision on NGS

0.92

final recall on NGS

Why a hierarchy beat a single model

One classifier facing 1:5,000 has to be a recall hero and a precision hero at once — impossible at that imbalance. Splitting the job let each stage do exactly one thing well:

Cheap before expensive — a keyword scorer clears ~90% of the corpus before any ML touches it, so the heavier models run on far smaller inputs.
Recall first, precision last — Steps 1–2 only ever remove confident negatives (NGS recall 0.99 → 0.95); precision is deferred to Step 3, where the ratio is finally tractable.
One number to watch per stage — NGS recall — which made the whole cascade easy to reason about and tune.

Running it over billions of documents

The whole pipeline was Dockerized and orchestrated on AWS Batch, fanning out across thousands of VMs. The funnel shape paid off operationally too: because each stage shrank the dataset, every more-expensive stage downstream ran on a fraction of what the stage before it saw.

← Back to overview Next: Extracting biomarkers →