Boo Bench is built to capture the pages that scare VLMs (cause them to make mistakes) but most specialized document OCR APIs perform well on. It is a playful and simple litmus test for understanding the need for specialized models, tools, and agent harnesses for document processing, beyond simply straight-shotting documents to VLMs.
A Boo is an element on a page that scares a parser. We frequently see failure modes where certain pages cause parsers to go into repetition loops or stop parsing properly.
document parsers and vision language models walked through all 179 rooms. Ghost-free is the share of pages they came back from with an answer and without a Boo. Coming back with nothing is a Boo too, and so is a page they were still looping on when the clock ran out.
Every page gets one question: did the output repeat anything more than the page has? The output is cleaned of markdown first (table separators and grids, rules, bold and strike markers, html tags, image links). A page is a Boo when any of these goes over:
Leaders, rules and blanks, any mix, counted in characters. One run may be at most 2× + 40 the page's longest, or one line of the page (250). All fill together may be at most 2× + 40 the page's, or one line per line of the page.
A run of one character, a short unit or a token may be at most 2× + 20 the times that unit occurs anywhere on the page.
A line repeated in the output may appear at most 2× + 40 the count of its rarest word on the page. The page's reading order can split a repeated row into columns, so the line itself is not counted. On a page whose transcription is exact (a generated ghost page), the line's own count is the budget.
Three real failures on mined pages, each checked against the rendered page.
Chosen from 131k pages with a repetition in their text layer, out of a 2.64M-page pool: one per document, near-duplicate templates removed, and the pages that came closest to tricking a parser kept, spread over 23 kinds.
15 scans of the UDHR's Article 1 with the alphabet and digits, in scripts that make VLMs loop. These pages repeat nothing, so any repetition in the output is a loop: the budget is 20 copies of any unit and no fill.
16 image-only pages of Halloween emoji, drawn together with their exact transcription, so the budget is exact too. A repeated row may appear at most 2× + 40 its count on the page, and anything the page lacks at most 20 times.
Each family is a way documents tempt a model to loop, with three Halloween pages drawn from an exact transcription. Open a page to see what a perfect parser writes.