Deep dive · 01

The problem — a needle in four billion documents

ConcertAI had licensed one of the richest oncology datasets in existence. The catch: the thing that made it valuable — genomic sequencing reports — was buried, rare, and unstructured. Finding it was the whole game.

What we had — a four-decade clinical data dump

4M patients every cancer type, many US health systems

~800–1,000 docs per patient, every healthcare interaction

~4B documents overwhelmingly unstructured text

A patient's journey is a long trail of documents — and almost none of it is in a clean, natural-language flow.

Detail — what's actually in the pile

Each document was generated from a real interaction with the healthcare system: nurse notes, phone-call logs, doctor visits, lab reports, imaging documents, pharmacy records, and billing/insurance paperwork. Much of it wasn't prose at all — think OCR of a tabular lab layout — and the database exposed only a few structured metadata fields on top of a mountain of raw text.

What we needed to find — and why it's gold

NGS panel reads mutation status of 500–1,000 biomarkers

Mutation profile which genes have mutated, and how

Trial eligibility which treatments & clinical trials a patient qualifies for

Cancer is a genetic disease. A Next-Generation Sequencing (NGS) report is the genomic readout that decides a patient's treatment options — which is exactly why pharma will pay for it.

Detail — why pharma cares so much

To run a clinical trial, a pharma company needs patients whose tumors carry a specific mutation. A structured, queryable set of NGS-tested patients lets them find and enroll exactly those cohorts — turning an un-searchable text dump into the most valuable kind of oncology data product.

The hard part — it's a needle, not a haystack

1 : 5,000

NGS reports per document at source

1–2

NGS reports per patient (if tested at all)

~30K

of 4M patients ever had an NGS test

Tech is recent NGS only became commercially available ~2010, picking up after ~2015.

Data is old The patient records span four-plus decades — most predate NGS entirely.

Thin recent slice Only a small, recent fraction of patients were ever sequenced.

So the target wasn't just rare in the abstract — it was structurally concentrated in a sliver of the timeline, with everything else as noise.

The old solution — and why it failed

Doctors keyword-search the raw text, then manually screen reports

Poor recall — NGS missed

Impossible to guess every right keyword combination, so real NGS reports slipped through and never made it into the product.

Poor precision — high screening-failure rate

Searches surfaced floods of false positives (insurance forms, doctor notes), text couldn't be queried on quantitative criteria, and every false hit cost expensive expert time — a high cost per patient found.

Manual keyword screening failed on both axes at once — and at 4B documents, more humans was never going to be the answer.

Detail — the brute-force cost

Done by hand, finding one NGS report meant opening, eye-checking, and closing on the order of 5,000 documents. Then, for each real report, an expert would have to transcribe the mutation status of 800–1,000 biomarkers. Across millions of patients, that's not a staffing problem — it's an impossibility.

The spec the solution had to hit

Algorithm spec

Drastically cut the screening-failure rate (precision)
Capture more NGS reports to grow product volume (recall)
Select patients on quantitative criteria
Work despite heavy noise in unstructured data

Engineering spec

Run over a massive payload: ~4M patients, ~1,000 docs each
Be cheap enough to run the whole database, repeatedly
Use only the text — no clean structured source to lean on

Everything in the next three pages exists to satisfy this spec: precision and recall on a 1:5,000 target, queryable structure pulled from noise, at four-billion-document scale.

← Back to overview Next: Finding the needle →