Datalab leads another competitor's extraction benchmark

LlamaIndex, another document parsing company, recently published ExtractBench, a 370-document benchmark for structured extraction, with an accompanying paper and public leaderboard. Its headline metric is a “unified value F1” — the share of leaf values in the target schema you get right, matched against a ground-truth JSON.

They scored Datalab at 64.5% — rank 13 of 14, near the very bottom.

That number was low enough to be suspicious, so we pulled the benchmark, ran it ourselves, and read the grader. Most of that gap turned out to be a bug in the benchmark’s Datalab adapter — not our extraction. Corrected, Datalab scores ~94%, behind only LlamaIndex (the creators of the benchmark).

Here’s what we found.

The bug: our confidence metadata was counted as wrong values

Datalab’s extraction API returns more than the values you asked for. Alongside each field it attaches a _meta sidecar — a small object with the field’s confidence score and a citation back to where in the document the value came from. A field named total_revenue ships next to a total_revenue_meta sidecar. It’s metadata about the extraction, not part of the extracted data.

Every extraction harness has to strip that kind of housekeeping key before grading, and ExtractBench’s Datalab adapter does — it has a routine that removes *_citations and *_score keys before the result reaches the scorer. It just doesn’t strip *_meta. So every confidence/citation sidecar we return survives into the graded object, gets lined up against the ground-truth schema (which has no such keys), and is counted as an extra, wrong predicted value.

The bug’s fingerprint is sitting on the leaderboard in plain sight. Datalab has two entries on ExtractBench: our API at 64.5% (13th), and lift, our open-source 9B extraction model, at 77.3% (10th). lift is a bare model — it returns the values you asked for and nothing else, none of the confidence-and-citation _meta sidecars our API attaches — so it hands the adapter nothing extra to miscount. The API carries that metadata and gets taxed for every field of it. The result is a tell you can read straight off their own board: it ranks our free model roughly 13 points above our paid API, on the same documents. LlamaIndex not even taking the time to do basic sanity checks on benchmark numbers tells you a lot.

The effect is easiest to see on a document we get perfectly right.

{
  "total_revenue": 1382.0,                              ✓ correct
  "total_revenue_meta": { "confidence": 0.98, "cite": … },   ✗ counted as a wrong value
  "gross_profit": 812.4,                               ✓ correct
  "gross_profit_meta": { "confidence": 0.97, "cite": … },    ✗ counted as a wrong value
  … 14 real KPIs, each with its _meta twin …
}

A byte-perfect extraction scored 0.588.

And the damage is systematic in a way that quietly skews the whole leaderboard. The _meta penalty is a roughly fixed tax — one junk key per field — so it hurts most on documents with few real values. Dense tables with thousands of cells barely notice it; sparse documents — single-page forms, a handful of scalar KPIs, tax documents — get cratered. ExtractBench is heavy on exactly those sparse documents, so the aggregate isn’t just noisy, it’s biased downward:

Same extraction quality, wildly different scores — sorted entirely by how much a fixed metadata tax gets to dominate.

How much does this cost us? The clean way to measure it is a controlled toggle: take the same extraction outputs and score them with _meta present versus stripped, changing nothing else. That lifts the score from 0.73 to 0.93 — and on documents we extract perfectly, it’s the entire gap (the earnings deck above: 0.588 → 1.000). Across the full 370-document set, teaching the adapter to strip _meta (exactly as it already strips _citations and _score) brings our corrected aggregate to 91.3%, versus the 64.5% LlamaIndex published.

The second bug: one vendor’s format becomes the definition of “correct”

Fixing the metadata bug gets us to 91.3%. The rest of the gap is subtler but still structural. Many values have more than one correct representation — a blank numeric field is 0 or null, an unchecked box is false or null, a thousand dollars is $1,000 or 1000, fifty percent is 50% or 0.5. The benchmark picks one of those surface forms for the ground truth (designed around how LlamaIndex returns the values) and scores everything else by exact string match:

It’s tempting to file these under “judgment calls.” They aren’t — because the grading isn’t neutral. The party that wrote the answer key also ships the system that tops the leaderboard, and exact-match scoring rewards whichever provider emits the answer key’s exact surface form. A system tuned to produce the ground truth’s conventions scores full marks; every competitor that normalizes differently is marked wrong for a formatting choice, not an error. That turns one vendor’s output format into the definition of “correct” — a structural penalty applied to everyone else, on cells where nobody is actually wrong. The unchecked-checkbox convention alone is 381 fields across 52 tax documents. Score these the way any consumer of the data would — a number is a number regardless of the currency symbol, a blank is a blank, an unchecked box is unchecked — and Datalab goes from 91.3% to 93.6%.

By group, the corrected number holds up across document lengths: short 94.1%, medium 92.2%, long 94.7%.

The usual caveats about vendor benchmarks

We’d give this the same treatment we’d give any self-published benchmark — including our own. ExtractBench was built by LlamaIndex, and LlamaIndex’s own product tops it. The party that ranks first also chose the corpus, wrote the schemas, and built the grader (including the adapter that scored everyone else’s output). They also used this for significant marketing value without even doing basic validation on the numbers. It’s just the structure of a benchmark a company publishes about itself, and it’s exactly why a metadata-stripping gap in the harness’s own Datalab adapter can ship without anyone noticing it was counting our confidence metadata as wrong answers.

A few honest notes on our numbers, too:

• Extraction is nondeterministic, so a single run wobbles by roughly ±0.5–1 point at the aggregate. 93.6% is one run; treat it as ~94%, not a decimal-precise claim.
• The _meta fix is one line in the harness’s Datalab adapter — strip _meta alongside _citations/_score — and it under-counts any provider that returns confidence metadata, not just us. Our current API still emits _meta, so the harness under-counts it today.

So take our corrected 94% for what it is: strong evidence on a benchmark we didn’t design, once a grader bug is removed. For your own decision, the only benchmark that counts is the one you run on your own documents. Point our playground at a few of your ugliest files, define what “correct” means for your workload, and trust what you measure over what any of us publish.