We lead an external extraction benchmark (but run your own evals)

Recently, a competitor of ours (Reducto) commissioned LongExtractionBench, a benchmark for pulling structured data out of long, dense tables. Reducto wrote the grading methodology, and when it launched they were the top performer. They chose recall as their headline metric, and scored Datalab at 33.8%. Recall here is defined as the share of a document’s rows a tool actually captures.

This benchmark tested an edge case that we weren’t well set up to handle. Like most vendor benchmarks, it was specifically designed for marketing - Reducto was intended to win, and structurally it is very hard for anyone else to match their score. The task, extracting thousands of rows from dense tables, is a valid extraction task, but only matters for a small fraction of customers.

Nonetheless, we took this as a challenge. We rebuilt our long-table extraction from the ground up - and today, on the public benchmark set Reducto released (50 documents), Datalab has the highest recall, ahead of Reducto itself.

For several reasons, we wouldn’t read too much into this benchmark, though (more below) - we’d encourage you not to trust vendor benchmarks (including ours!), and to run your own evals.

We now lead on recall and precision

Here’s where things stand today on the born-digital set:

A few notes:

• Datalab and Reducto are a fresh, same-corpus re-run. We ran both on the same 50 born-digital documents with the official grader, with cache-busting so neither system could serve a stored answer. All three metrics land within a point — these systems are very close.
• Datalab doesn’t win every metric. We lead recall (99.1 vs 98.3) and precision (99.8 vs 99.7); Reducto edges cell-level (leaf) accuracy by ~0.7 (99.8 vs 99.1). We report all three rather than just the one we win.
• On recall, Datalab also leads the rest of the field — Extend 92.7%, LlamaParse 77.5%, Gemini 3.1 Pro 48.6% (Reducto’s published figures on the full 225-document set, which includes documents we can’t see; close to what we measure, but not the identical corpus).

The task it measures

LongExtractionBench tests one specific thing: taking a document with a very long, very dense table — up to ~15,000 rows over hundreds of pages — and extracting every single row into a JSON schema.

Most of the failures in this mode are recall failures. It’s relatively easy to map table rows to the schema, but doing it perfectly across thousands of rows is challenging. It’s even harder if some rows should be included, but not others.

[
  {
    "timestamp": "09:14:00",
    "channel": "CH-07",
    "reading": 0.482,
    "unit": "V",
    "flag": "ok"
  },
  … 14,207 more objects
]

A note on vendor benchmarks

LongExtractionBench is not a general test of document extraction. It’s built around a single edge case — thousands of near-identical rows in dense tables — that most workloads never hit, and that happens to be a Reducto strength. Most documents aren’t 15,000-row tables; most extraction jobs are a few dozen fields off an invoice, a form, a contract, or a report, where reading order, layout, handwriting, and language matter far more than raw row throughput.

The leader also scores 99+%, which is a red flag that either the benchmark has been fit to the specific vendor (tailored to their strengths), or is relatively easy. Scores on more realistic benchmarks tend to max out at lower scores, with headroom for improvement.

Here are some factors to pay attention to in vendor benchmarks:

Right in the benchmark’s limitations section: “Reducto commissioned this benchmark. Reducto is also one of the systems under test and the top performer on these documents.” The same party that ranks first also chose the task, commissioned the set, and wrote the grading methodology. This is true for any vendor benchmark, including our own.

So take our recall win for exactly what it is: strong evidence on one hard, narrow task, on a benchmark we didn’t design. Here’s how we recommend you benchmark document extraction systems:

1. Collect 20–50 documents that look like your real workload
2. Define what “correct” means for you. Every row of a table? A handful of header fields? Reading order? Non-English text? The answer is workload-specific, and it’s rarely a single composite score.
3. Run every candidate the same way, on the same inputs, and read the actual failures — not just the aggregate.

We try to be as objective as possible with our own benchmarking - use external benchmarks where possible, if we have internal benchmarks make them fair and representative of difficult real-world tasks. But you should still run evals on your own documents. We built Forge Evals for exactly this, and our playground is easy to do some quick testing with. Also, feel free to reach out to us if you have custom eval needs. Use what we offer, use a competitor’s tools too, and trust what you measure over what any of us publish.