6 articles tagged “evaluation”.
Our held-out registry knew every evaluation page and refused to train on any of them. It knew nothing about the other pages that print the same text: a commentary quoting the passage it annotates, a variant in another volume, an encyclopedia quoting a chronicle. An eight-word shingle check found 25 of them in the harvest, and nothing had failed
We set out to lead every Armenian OCR engine we can measure on every register. Before training anything, we found that most of the gap we were chasing was in our own metric. The rest came down to reading order, a word list and a second fine-tune: v6 now leads both means and most registers outright
We measured two compact vision-language models as candidate local backends. One is in. The other produced the best score any engine has recorded on our hardest fixture, and it did it by inventing words that were never on the page
Three weeks ago we published a cross-engine comparison with one row missing, because the model had gone and we would not guess at it. We have now downloaded it and run it over every register. It takes the mean off us, and it takes every register the previous table had given to Calfa's Tesseract model
Until now, every cross-engine number we had published for Armenian came from ten encyclopedia pages. We have now run the other engines across all eight of our evaluation registers. Our model still leads on average, but it loses one register in three, and two things we thought we knew about where the gap lives turned out to be wrong
Two days after shipping v3 we built the evaluation that showed what it could not do: read anything but the encyclopedia it was tuned on. This is the story of v5 — a corpus three times wider, a filename bug that taught a model to hedge, and the first version to read Western Armenian at all