Open benchmarks·DOCX tooling·Legal AI·MMXXVI
Contracts and briefs pass through editors, converters and AI assistants that each claim to handle the work. benchmarks.law puts them on the same documents and tasks and checks the result against a reference: Microsoft Word for the file, experienced lawyers for the law. The DOCX benches publish every input, output and log; the legal-AI studies publish their method and every tool's scores.
Each editor's own code imports a purpose-built adversarial document and exports it again. The output is scored construct by construct and opened in Word. A separate set of ten hostile packages tests what each editor refuses. Eight reports so far.
Every engine converts the same corpus to PDF, or produces a tracked-changes document from the same pair, and each page is rendered beside Word's and scored by pixel, ink overlap, drift-tolerant SSIM and text boundaries. Click any case; overlay any two.
Accuracy: tasks that met the accuracy standard, of 18
On 18 real in-house extraction tasks, the most accurate of six AI assistants was not a legal tool, and the two legal tools gave the most useful answers. Lawyers graded all 108 answers blind; no model graded anything.
Anna Guo, Arthur Souza Rodrigues · legalbenchmarks.ai · Human-judged · 2–4 lawyers
Output reliability: drafts that passed, of 30
On 30 real drafting tasks, nine of thirteen AI tools were at least as reliable as experienced in-house lawyers, and legal AI drafted no better than general AI. 450 outputs, 72 survey responses, 12 interviews.
Anna Guo, Arthur Souza Rodrigues, Mohamed Al Mamari, Sakshi Udeshi, Marc Astbury · legalbenchmarks.ai · Mixed · LLM jury + human review
This edition does what the previous one could not: it downloads every editor's code, runs each editor's own import→export path over a purpose-built adversarial document, and scores the result. Every cell that previously read NOT VERIFIED IN CODE is now either a source citation or a measurement.
Report AA companion to Open-Source DOCX Editors, Re-verified Against Code and Measured by Round-Trip (2026-11-01), and the first half of what used to be Report A. That report asked how faithfully each editor preserved a document, and answered with a single number. This one opens the question up: a DOCX file is not one thing but twenty-seven vocabularies, and an editor does not preserve or lose "a document" — it preserves or loses each vocabulary, in its own way, for its own reason.
Report A2A companion to Open-Source DOCX Editors, Re-verified Against Code and Measured by Round-Trip (2026-11-01), and the second half of what used to be Report A. The first report asked how faithfully each editor preserved a document and answered with a number. Serialization by Namespace opened the question up vocabulary by vocabulary. This one asks the complementary question: of the bytes that came out, how many did the editor never even look at?
Report BA companion to Open-Source DOCX Editors, Re-verified Against Code and Measured by Round-Trip (2026-11-01) and to What Is Serialized, What Is Pass-Through, and What Is Actually Rendered (2026-10-30).
Report CPart C of the series. Part A covers serialization by namespace, Part A2 covers pass-through. This part answers one question: when you open the corpus in each editor's own running application, what does that application put on the page — and how does it compare with what Microsoft Word puts on the page?
Report DNo editor is safe on every package. Safe here means the weapon is not in a finished document and the process did not hang or run out of stack. A named ceiling is stronger than a bare refusal. A bare refusal is stronger than a hang. Keeping the weapon is not safe. Word was not opened in the seatbelt matrix. A separate Word run is in Report D2 - What Word Protects Against (2026).md, final since 2026-10-31; its results are copied at the end of this report.
Report D1Jubarte is not one of the editors in Report D. This file is only the release binary of ../jubarte-redlines.
Report D2This report records what Word does with each hostile package: whether it opens it, warns, refuses, hangs or crashes, how much memory it takes, and whether it reaches the network. It is the reference the editors in Report D are compared against when we say a refusal "costs no Word-valid input".
Each tool is driven through the import and export its own project documents. No harness rewrites its output, and a step that fails before the tool runs is recorded as not run, not as a result.
Word opens every output and draws the page that renderings are scored against; on the DOCX → PDF bench Word's own export is the oracle. A file Word refuses to open is reported as refused, not repaired and re-scored.
Inputs, outputs, page images, scores and the full logs sit beside every table. The maintainer also writes Jubarte, one of the measured tools; its rows are marked and follow the same rules.