The research

Tetrak OCR exists because of a benchmark, not the other way round. This is that work end to end: why we ran it, what we measured, what it found, the one finding we got wrong, and the four mechanisms the evidence forced into the product.

Read it in order and it is an argument. Jump around with the contents on the right and it is a reference.

Why run a benchmark at all

A real archive is not a stack of uniform pages. The nine items in this corpus include a linen postcard, a chromolithograph poster, two pages of 1930 newsprint, a bilevel TIFF of a theatre playbill and a twenty-one-page souvenir programme. They differ in typography, contrast, layout and physical condition, and they were digitised by different people at different times.

The obvious way to build an OCR tool for that material is to pick a good engine, configure it well, and run it over the batch. Every vendor comparison, every “best OCR library” article, and most of our own instinct pointed that way.

We could not find evidence that it was the right shape. So before designing anything we built a fixed corpus, generated reference transcripts, and measured every engine we could install against every document. The result decided the architecture — and, in one case, unbuilt a feature that had already shipped.

What we measured

Nine items of Los Angeles stage and picture-palace ephemera, c. 1910–1945, chosen to span the failure modes that matter in archive digitisation rather than to flatter anything. Provenance and rights for every item are in the corpus reference.

Each transcript is scored on two numbers, both against normalised text — lowercased, whitespace collapsed.

Character similarity compares the transcript to the reference as a sequence. It is order-sensitive: text read in the wrong order scores badly even when every word is present.

Word recall is the share of expected words appearing anywhere in the transcript. It is order-insensitive, and it ignores extra words — an engine that invents text is not penalised.

They are always reported as a pair, because they fail differently and the gap between them is where the interesting behaviour lives:

On this corpus that is not a theoretical distinction. It is the finding.

The weakness at the centre of the method

Reference transcripts were generated by Claude (claude-opus-4-8) and are committed under evaluation/corpus/expected/.

The reference is committed rather than regenerated per run precisely because of that instability; a moving reference would make run-to-run comparison meaningless. Human-checked ground truth would fix this properly, and it is the single highest-value improvement available to this work.

What the numbers support: relative comparison between local engines on this kind of material, and relative speed. The metrics are deterministic, the corpus is fixed and public, the reference is committed, so the local columns reproduce exactly.

What they do not support: any claim about Claude’s absolute accuracy, or generalisation beyond archival print of this era and condition.

The results

Character similarity / word recall, nine fixtures, run on an Apple MPS machine so auto-local had Marker available. Bold marks the best local engine per fixture; N/A means the engine cannot read PDFs.

Generated from evaluation/benchmark.csv at build time. Bold marks the best local backend per fixture. claude generated the ground truth and is a ceiling reference rather than a score.

Generated from evaluation/benchmark.csv at build time. claude generated the ground truth and is a ceiling reference rather than a score.

Both the table and the chart are generated from evaluation/benchmark.csv at build time, so neither can drift from the data.

What the numbers say

No engine wins across the board

This is the headline, and it is why the average column is the least useful part of the table. Five different engines win across the nine fixtures:

• Marker takes the newsprint cover (0.39), the Kinema ad (0.75) and the PDF (0.66)
• Retuned Tesseract takes the postcard reverse (0.91) and Grauman’s (0.86)
• PaddleOCR takes the playbill (0.39) and the poster (0.13)
• EasyOCR takes the Carthay Circle front (0.87)
• Plain Tesseract takes the interior newsprint page (0.45)

An earlier version of this page added “and none wins more than twice”. That is no longer true — Marker takes three, both TIFFs having landed in territory it handles well. The claim that matters is unchanged: no engine wins most of them, and the winner is not predictable from the average.

Route by document, not by average.

The metrics disagree, and that is the point

On newsprint every local engine pairs high word recall (0.71–0.94) with near-zero character similarity (0.03–0.45). The words are all there; the columns are interleaved. Character similarity is order-sensitive and word recall is not, so the gap between them is the layout failure, quantified.

PaddleOCR recovers 94% of the words on the Inside Facts front page and scores 0.21 on similarity. For a human reader that output is unusable; for full-text search it is adequate. Which number matters depends on whether you are building a reading edition or a search index, and that is your question rather than ours — which is why we never collapse the two into one score.

Tuning beat switching

tesseract-auto averages 0.41/0.63, ahead of EasyOCR at 0.35/0.60 — a deep-learning engine beaten by Tesseract with better per-image settings.

It has not always been. It previously scored 0.22 and lost to plain Tesseract. The fix was re-fitting the page-segmentation bands in analyse_image(), which had been inherited from a parent project and were sending dense pages to the sparse-text mode. Carthay Circle’s front went 0.00/0.00 → 0.49/0.86 on that change alone.

The same engine, the same image, a different page-segmentation mode. Reaching for a heavier engine is the expensive reflex — gigabytes of weights, minutes of runtime, sometimes an API bill — and on this evidence it is often the wrong first move.

…but the tuning does not generalise

Two TIFFs were added after analyse_image()’s thresholds were fitted, which makes them the only held-out data here. Auto-configuration is worth nothing on either:

On the fitted seven, auto-configuration gains on five fixtures, once by +0.49. On the two it has never seen it gains nothing and loses badly on one. That is what overfitting looks like, and with seven images against four contrast bands and three PSM bands there was ample freedom for it to happen.

The Kinema failure is diagnosable rather than mysterious:

The playbill lands exactly on the defaults, so auto-configuration is a no-op. The Kinema ad is 42.9% dark because a large illustration fills half the frame, which puts it in the “image-dominated, no page structure worth analysing” band and selects PSM 6, a single uniform block. But it does have structure: a narrow programme column in tiny type beside the headline block. PSM 3 reads it at 0.23; PSM 6 collapses it to 0.04.

analyse_image()’s own docstring anticipated the risk — “it cannot tell ink from imagery” — and reasoned that a dark frame has little structure to analyse anyway. That held for the night-scene postcards the band was fitted against. It does not hold for an illustrated page that still has columns, and a histogram cannot tell those two apart.

Read 0.49 as an upper bound rather than an expectation on new material. Re-fitting the bands against a wider set is planned; this is the evidence for why it matters.

Note what did not happen: adding two fixtures changed no other engine’s ranking, and auto-local still routed the Kinema ad correctly by picking Marker at 0.75. The quality score caught what the heuristic got wrong — which is the argument for measuring transcripts rather than predicting from pixels.

Fan-out wins, by less than it should

auto-local averages 0.53/0.74, the best local result on both metrics, and its margin over tesseract-auto widened from 0.05 to 0.12 as the corpus grew, because fan-out measures each transcript instead of predicting from pixels.

The ceiling is higher. An oracle taking the best local engine on every fixture averages 0.60. auto-local reaches 0.53, so the quality heuristic captures roughly half of what fan-out makes available. Four fixtures account for the gap:

The Carthay Circle case repays reading closely, because the raw quality score got it right and the ranking then overrode it. EasyOCR scored 0.1416 against Tesseract’s 0.1149 — but Tesseract emitted 77 words to EasyOCR’s 11, and the winner is ranked on quality × (0.5 + 0.5 × words / max_words). That multiplier exists to stop a conservative engine with a few clean words beating a more complete one. Here it did the reverse, promoting a transcript scoring 0.49 over one scoring 0.87.

Improving the selection is the cheapest accuracy work available: it is a scoring weight, not an engine limitation.

A feature we measured and then removed

auto-local-fast ran only the top-ranked eligible engine instead of all of them. On this machine that was always Marker, so it produced identical scores to marker (0.42/0.61) at almost identical cost — strictly worse than an engine already available:

Lower accuracy and twenty-six times the runtime. Nor was that only a GPU-machine problem: worked through every installed configuration, it never won. tesseract-auto already occupied the niche fast mode was invented for and filled it better, so the backend was removed, one day after it shipped.

Registering it as a backend rather than hiding it behind a flag is what turned a preference into a measurement, and then into a deletion.

Marker is uneven and slow

It wins the newsprint cover, the Kinema ad and the PDF, then collapses on Grauman’s (0.09/0.00), where its layout model reads the whole postcard as a figure and returns nearly nothing.

It also dominates the runtime. Of the roughly 65 minutes a full --all run takes, about 50 are Marker. Dropping it from the fan-out pool would cost more than it saves: it wins three of nine, including the Kinema ad at 0.75 where the next best local result is plain Tesseract at 0.23.

The poster defeats everything local — and destabilises the reference

Best local character similarity is 0.13; retuned Tesseract manages 0.02.

Claude is the only thing that reads it, but how well is not a stable number. Across four runs against the same committed ground truth — which Claude itself generated — it scored 0.73, 0.24, 0.26 and 0.73. Word recall over the same four runs was 0.98, 0.96, 0.97 and 0.98.

That split is the finding. Claude recovers the same words every time and arranges them differently every time, because a chromolithograph has no single correct reading order for a transcript to be scored against. It is the same signal as the interleaved newsprint columns, arriving for a different reason.

Hand-lettered display type is where matching letter shapes stops working and understanding what the page is starts mattering. If your material looks like this, the local tools will not do — and this benchmark cannot tell you precisely how much better the alternative is.

A finding we retracted

An earlier version of this analysis claimed auto-local mis-ranked the PDF, choosing Marker when Marker had scored 0.02 character similarity there.

That was wrong and has been retracted. The 0.02 was an artefact of truncated ground truth — the Claude backend was calling the API with max_tokens=4096 and silently returning partial transcripts for the twenty-one-page programme, so Marker’s correct output was being compared against a fragment. Against the corrected reference Marker scores 0.66 and is the best engine on that fixture. The quality heuristic had been right all along.

It is recorded rather than quietly deleted because the failure mode is instructive: a truncation bug in the reference looked exactly like a routing bug in the system under test. The Claude backend now raises on truncation rather than returning partial text.

What the evidence built

Four findings, four mechanisms. Together they are auto-local, and together they are the six stages between dropping a file in and getting a transcript out.

Fan out, and select per file

No engine wins, so the obvious design — pick one, configure it, apply it to the batch — is unavailable. Any single choice is wrong for most of the corpus.

auto-local runs every viable engine over each file and keeps the best transcript. Selection happens per file, not per batch, which is the only granularity at which the finding can be acted on. The cost is honest: several engines per file, so it is the slowest local mode.

The engines run sequentially, deliberately. Each is heavy on CPU and memory and several hold model singletons; starting two or three at once trades predictable runtime for contention, and it hurts most on exactly the large batches where the saving would matter.

Auto-configure before reaching for a bigger engine

Tuning beat switching, so analyse_image() inspects each image before OCR and picks contrast and page-segmentation mode from two cheap statistics:

The second misleads by its name. dark_pct looks like a text-density measure and is not: a night-scene postcard reads 56% dark because of the photograph. That turned out to be useful anyway — a frame dominated by imagery has little page structure for layout analysis to work with, however the darkness got there. The heuristic works for a slightly different reason than the one it was reached for, which is also why it fails on the Kinema ad above.

The exact bands are in the routing reference.

Score without a reference

The metrics disagree, and both need ground truth. At run time there is none, so whatever picks the winner has to judge a transcript it has never seen the answer to.

Two signals that fail differently, multiplied, then weighted by relative word count:

• Dictionary coverage — the share of tokens that are real words. Catches character-level garbage like e1rr0r.
• Perplexity, via GPT-2 — how natural the text reads. Catches the real-word substitutions a spell check waves through.

The word-count weighting came straight out of an observed failure: on sparse images a conservative engine emitting eight clean words was beating one that recovered twenty-three noisier ones, because dictionary coverage is trivially 1.00 for a short clean phrase. Quality alone rewarded saying less. As the Carthay Circle case shows, that correction now overshoots in the other direction on at least one fixture.

Fail loudly

Some pages defeat everything local. For a tool running unattended over an archive that is the dangerous case — not because it fails, but because a failed transcript looks exactly like a successful one from outside. A .md file appears in processed/, and nothing signals that it is noise.

So there is a quality floor of 0.10. Below it the scan is diverted to triage/ with a manifest recording what every engine produced and how each scored. It never reaches processed/.

The point is not that the pipeline handles hard documents. It is that you can tell which ones it did not, which makes “what is still in triage/” a meaningful question and an actionable queue.

What we chose not to build

No automatic escalation. The triage queue is passive. Forwarding failures to a vision model automatically would be easy, and would turn a local, free, offline tool into one making paid API calls on material the user may not have wanted to leave the machine. That decision stays with a person.

No single quality number. Character similarity and word recall are reported as a pair everywhere, because collapsing them hides the finding above.

No claim about Claude’s accuracy. It generated the ground truth it is scored against. It sets a ceiling reference and nothing more.

Choosing a tool for your own material

The practical version of “no engine wins”:

On the Carthay Circle postcard reverse, five points of character similarity separate a free offline tool (tesseract-auto, 0.91) from a paid API call (Claude, 0.96). For clean printed text that is the norm rather than the exception, and it is the case where reaching for a vision model is hard to justify.

If you do route decorative and multi-column material to Claude, a page runs roughly 1,500 input and 750 output tokens on this corpus:

A practical pattern: run auto-local over everything and send only what lands in triage/ to Claude. On this corpus that would be the poster and little else.

What is still open

• The selection heuristic loses 0.07 average against an oracle that knows the answer. That is a scoring weight, not an engine limitation, and it is the cheapest accuracy work available.
• The auto-configuration bands are overfitted to seven of nine fixtures and need re-fitting against a wider set.
• Ground truth is machine-generated. Human-checked references would remove the benchmark’s central weakness.
• Nothing here generalises beyond archival print of this era, condition and script. Whether the pipeline works on other scripts is untested — an open Armenian Tesseract model exists and would make the cheapest first experiment.

Reproducing all of this

pip install -e '.[all]'
tetrak-ocr evaluate --all --save

Expect roughly 65 minutes, most of it Marker. Results land in evaluation/benchmark.{csv,md} with a dated copy in evaluation/runs/.

Local engines reproduce exactly between runs; the Claude column does not, for the reason given above. Compare like with like by re-reading the dated CSVs rather than trusting a single pass.

Full detail lives in the reference section: the corpus with provenance and rights, the engines with each one’s constraints and failure cases, and routing with the decision table and bands.