bench 1.0.0 · generated 2026-11-03

Scored against Word, failures included.

The oracle is Word’s own PDF export of each document (SHA-pinned). A converter’s PDF is scored 0–100 by pixel similarity; redlines are additionally opened in Word and compared against Word’s own compare. Medians and means count a failed document as 0, so no tool gains by skipping a hard one. Every table is a ranking, and the tool that leads it is named above the table.

Accepted and randomized redline DOCX to PDF vs Word export

Document set `c28442e9bb5a5deb` (400 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.8`.

Each converter turns accepted and randomized-redline DOCX documents into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same file. A failed conversion scores 0 for that document (intent-to-treat).

Leader: jubarte 0.12.0 — median 82.45

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte 0.12.0
0.12.0@3c81203f1f66
[81.13, 83.23]
82.45 79.93 400 0
1=
jubarte 0.11.0
0.11.0@838d0e9603c8
[77.54, 81.89]
79.75 76.24 400 0
3
Wordcraft CLI 0.7.0
wordcraft-cli 0.7.0@45fa52a260fb400857c6033c3cbfa90f8e9c67d05aeb9417cccbb3d6c51c0960
[52.41, 55.60]
53.82 56.59 400 0
3=
Wordcraft CLI
0.6.1@b71aef4ee85a
[51.41, 54.53]
52.82 55.46 400 0
3=
docxide-pdf 0.18.4
docxide-pdf 0.18.4@503ea7e085ae9acd16a4f263c7802325cad254eedeff4cb2509e88714019e00b
[50.63, 54.45]
52.37 55.90 400 0
3=
libreoffice_convert_rust 0.1.0
libreoffice_convert_rust 0.1.0@99552eb9ead6dc5dd2bbb3c69cbed958888dbb037553d586f08b5aa80794e172
[47.43, 48.97]
47.87 54.82 398 2
3=
soffice
LibreOffice 26.8.1.1 bf6ef2662401c7d6dce671ab4deb26ee0af3a7c1
[47.43, 48.97]
47.87 54.82 398 2
8
rdocx 0.16.0
rdocx-cli 0.16.0@8c4d40caffb12177ac8ab21d568a5dd6c389f9cb4bd742653c2b180102d0abee
[45.14, 47.11]
45.90 50.69 397 3
9
office2pdf 0.8.1
office2pdf-cli 0.8.1@ec266239b0bdbf71275e79fde2ffaaa1155321050d716eab45ef43f60d6ff9f9
[42.81, 45.81]
44.45 48.66 398 2
10
pdfitdown 4.2.0
pdfitdown 4.2.0@6a8275fe29d0c7fd61465f56ca55c21fc22c84fbd228ceef4288bfb28bceac36
[40.89, 43.34]
41.95 43.48 397 3

Source DOCX → PDF, no redlines

Document set `c947065c59b2` (398 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.6`.

Each converter turns the clean source DOCX documents, with no tracked changes in them, into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same source. A failed conversion scores 0 (intent-to-treat).

Leader: office2pdf — median 84.34

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
office2pdf
office2pdf 0.6.7
[79.83, 86.03]
84.34 73.82 390 8
1=
pdfitdown
pdfitdown 4.0.0
[79.83, 86.03]
84.34 73.82 390 8
3
libreoffice_convert_rust
LibreOffice Convert Rust v0.1.0
[72.14, 79.47]
75.26 73.98 398 0
4
jubarte
jubarte 0.7.0
[66.94, 68.10]
67.39 66.25 398 0
4=
docxide-pdf
docxide-pdf v0.17.0
[63.27, 64.73]
63.97 65.65 398 0
6
rdocx
rdocx 0.7.0
[53.38, 55.31]
54.49 59.11 398 0
7
dxpdf
dxpdf 0.5.1
[52.15, 52.94]
52.52 56.73 381 17

Every corpus/word DOCX → PDF vs Word's own PDF

Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `1.0`.

Word's PDF export of each DOCX in all four corpus/word states (clean; tracking without comments; comments with tracking; comments clean) is the oracle; the candidate is each converter's PDF of the same file, scored pixel-wise. A failed conversion scores 0 (intent-to-treat).

Leader: jubarte — median 82.36

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0+c607894
[82.19, 82.54]
82.36 80.12 6,427 0

The same documents under docxide-pdf's own metrics

Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `1.0`.

The same documents as the corpus-wide table, scored instead with docxide-pdf's own ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against the same Word PDF exports, so no converter is measured only by this bench's pixel scorer.

Leader: jubarte — median 70.68

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0+c607894
[70.20, 71.23] · text 100.00
70.68 66.84 6,397 30

Below 3 pages: every converter on the same short documents

Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.7`.

The corpus-wide conversion benchmark restricted to the documents Word lays out in 1 to 3 pages, so a converter that renders only the first 3 pages (unlicensed PyMuPDF Pro) is compared with every other converter on the same documents. The oracle stays Word's own PDF export of each document, scored pixel-wise; a failed conversion scores 0 (intent-to-treat).

Leader: jubarte — median 83.15

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.9.3
[83.01, 83.28]
83.15 81.35 2,626 0
2
soffice
LibreOffice 26.8.0.3 bce0998afefdbc355585ca324285661a2170ba77
[67.84, 68.83]
68.40 66.26 2,611 15
3
PyMuPDF Pro
pymupdf-pro 1.28.2 (unlicensed, first 3 pages)
[59.09, 60.77]
60.06 60.02 2,601 25
4
docxide-pdf
docxide-pdf v0.17.1
[53.81, 54.89]
54.24 56.00 2,626 0

Below 3 pages: docxide-pdf's metrics on the same short documents

Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `0.7`.

The docxide-pdf metrics on the same 1-to-3-page documents: each converter's PDF scored with ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against Word's own PDF exports of the same files.

Leader: jubarte — median 74.34

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.9.3
[73.30, 75.08] · text 100.00
74.34 68.08 2,626 0
2
soffice
LibreOffice 26.8.0.3 bce0998afefdbc355585ca324285661a2170ba77
[23.36, 24.50] · text 99.58
24.03 28.66 2,611 15
3
PyMuPDF Pro
pymupdf-pro 1.28.2 (unlicensed, first 3 pages)
[20.96, 22.71] · text 36.36
21.86 26.77 2,595 31
4
docxide-pdf
docxide-pdf v0.17.1
[11.17, 11.96] · text 24.32
11.42 19.93 2,619 7

Redlines vs Word's compare

Each tool redlines a document pair; Word opens that redline and exports it to PDF, and the PDF is scored with the pixel scorer against Word's own compare of the same pair. A pair with no scored PDF — no redline, or one Word could not open — counts 0 in the ITT columns; the = 100, ≥ 90 and < 50 counts and the docxide-pdf Ink Jaccard and Text boundary means are over the scored PDFs only.

Leader: jubarte-0.17.0 — median 84.56

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte-0.17.0
jubarte 0.17.0 (release)
= 100: 342 · ≥ 90: 729 · < 50: 500 · ink 0.75 · text 0.82
84.56 75.81 3,454 48
2
jubarte-0.14.0
jubarte 0.14.0 (release)
= 100: 344 · ≥ 90: 704 · < 50: 546 · ink 0.74 · text 0.81
84.46 74.33 3,414 88
3
jubarte-0.10.1
jubarte 0.10.1 (release)
= 100: 241 · ≥ 90: 615 · < 50: 619 · ink 0.73 · text 0.80
84.15 74.77 3,496 6
4
jubarte-rust
jubarte 0.10.0 (86b6b5d3)
= 100: 310 · ≥ 90: 658 · < 50: 793 · ink 0.67 · text 0.77
81.73 72.23 3,499 3
5
jubarte-093
jubarte 0.9.3 (release)
= 100: 309 · ≥ 90: 634 · < 50: 780 · ink 0.67 · text 0.76
79.07 68.62 3,338 164
6
docxodus-15.0.0
docxodus 15.0.0 (npm/WASM, set redline_600)
= 100: 46 · ≥ 90: 98 · < 50: 135 · ink 0.65 · text 0.71
75.70 64.86 543 57
7
docxodus
Docxodus 12.6.5 (C#)
= 100: 204 · ≥ 90: 482 · < 50: 614 · ink 0.65 · text 0.73
62.87 53.94 2,612 890
8
apryse
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked)
= 100: 0 · ≥ 90: 39 · < 50: 1285 · ink 0.18 · text 0.16
43.17 37.50 2,524 978
9
apryse-12.2.1
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked; set redline_600)
= 100: 0 · ≥ 90: 4 · < 50: 190 · ink 0.18 · text 0.14
37.87 28.42 336 264
10
superdoc-2.25.0
superdoc-sdk 2.25.0 (set redline_600)
= 100: 1 · ≥ 90: 1 · < 50: 3 · ink 0.21 · text 0.20
0.00 6.92 67 533
11
superdoc
superdoc-sdk 2.16.0
= 100: 4 · ≥ 90: 7 · < 50: 20 · ink 0.21 · text 0.29
0.00 6.66 380 3,122

Redlines accepted or rejected by Word

Each tool redlines a pair; Word accepts — or rejects — every change in that redline and exports the result to PDF, scored with the pixel scorer against Word's own compare of the pair resolved the same way. A pair with no scored PDF counts 0 (intent-to-treat); the calibration row is Word's own compare against itself.

Leader: word-identity — median 100.00

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
word-identity
Word's own compare (calibration) · accept all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
docxodus
docxodus 12.6.4 (npm) · accept all
= 100: 166 · ≥ 90: 299 · < 50: 16 · ink 0.85 · text 0.96
99.87 90.07 394 6
3
jubarte-rust
jubarte 0.9.3 (673aff74) · accept all
= 100: 181 · ≥ 90: 261 · < 50: 33 · ink 0.80 · text 0.94
99.70 87.56 400 0
4
superdoc
superdoc-sdk 2.15.0 · accept all
= 100: 4 · ≥ 90: 52 · < 50: 1 · ink 0.95 · text 0.98
0.00 13.95 58 342
1
word-identity
Word's own compare (calibration) · reject all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
jubarte-rust
jubarte 0.9.3 (673aff74) · reject all
= 100: 186 · ≥ 90: 260 · < 50: 35 · ink 0.77 · text 0.91
99.85 86.22 400 0
3
docxodus
docxodus 12.6.4 (npm) · reject all
= 100: 155 · ≥ 90: 275 · < 50: 26 · ink 0.80 · text 0.94
99.80 85.38 385 15
4
superdoc
superdoc-sdk 2.15.0 · reject all
= 100: 1 · ≥ 90: 36 · < 50: 1 · ink 0.83 · text 0.95
0.00 11.07 49 351

600-document samples

the release samples, scored on the same rules

DOCX → PDF vs Word's own PDF — 600-document sample

600 docs · 150/150/117/183 by state · scorer numpy · 2026-11-01

A state-balanced 600-document sample of corpus/word (every path and sha256 in release_info/sample_conversion_0.17.0_11-01-26_01-06.csv), each converter's PDF scored against Word's own export of the same file. Sorted by ITT median; a failed document scores 0. 95% CI is a percentile bootstrap of the ITT median (2,000 resamples, seed 42).

Leader: jubarte — median 81.09

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte
jubarte 0.17.0
[80.42, 81.7]
81.09 79.24 600 0
2
soffice
LibreOffice 26.8.0.3
[47.53, 48.47]
47.88 54.61 600 2

Redlines vs Word's compare — 600-pair sample

600 pairs · one Word compare each · pixel scorer · opened in Word · 2026-11-01

A sample of 600 document pairs of redlines_0929_full, one Word compare each: every pair of the scarce states, the rest spread over the document families (every path and sha256 in release_info/sample_redline_0.17.0_11-01-26_01-06.csv). Each tool redlines the pair; Word opens that redline and exports it to PDF, scored against Word's own compare. A pair with no scored PDF counts 0.

Leader: jubarte-0.17.0 — median 83.76

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte-0.17.0
jubarte 0.17.0
= 100: 47 · ≥ 90: 109
83.76 74.36 600 7
2
docxodus
Docxodus 12.6.5 (C#)
= 100: 38 · ≥ 90: 76
70.27 64.92 600 25

By corpus state

median · documents per state in brackets

clean (150)

jubarte90.77
soffice68.83

tracking, no comments (150)

jubarte81.54
soffice64.40

comments, clean (117)

jubarte81.22
soffice46.33

comments + tracking (183)

jubarte78.60
soffice42.85

clean (100)

jubarte 0.12.090.78
jubarte 0.11.090.72
Wordcraft CLI 0.7.068.29
Wordcraft CLI63.87
docxide-pdf 0.18.466.93
libreoffice_convert_rust 0.1.069.33
soffice69.33
rdocx 0.16.062.29
office2pdf 0.8.154.42
pdfitdown 4.2.044.02

tracking, no comments (100)

jubarte 0.12.082.16
jubarte 0.11.081.85
Wordcraft CLI 0.7.059.18
Wordcraft CLI55.56
docxide-pdf 0.18.450.25
libreoffice_convert_rust 0.1.065.47
soffice65.47
rdocx 0.16.049.35
office2pdf 0.8.145.48
pdfitdown 4.2.041.13

comments + tracking (100)

jubarte 0.12.080.11
jubarte 0.11.071.20
Wordcraft CLI 0.7.050.58
Wordcraft CLI48.46
docxide-pdf 0.18.446.71
libreoffice_convert_rust 0.1.042.12
soffice42.12
rdocx 0.16.041.75
office2pdf 0.8.140.73
pdfitdown 4.2.040.84

comments, clean (100)

jubarte 0.12.080.63
jubarte 0.11.068.97
Wordcraft CLI 0.7.051.94
Wordcraft CLI51.82
docxide-pdf 0.18.456.18
libreoffice_convert_rust 0.1.045.33
soffice45.33
rdocx 0.16.045.21
office2pdf 0.8.144.45
pdfitdown 4.2.044.23

clean (2,480)

jubarte89.84

tracking, no comments (3,410)

jubarte81.11

comments + tracking (420)

jubarte78.19

comments, clean (117)

jubarte81.22

clean (2,480)

jubarte81.13

tracking, no comments (3,410)

jubarte66.44

comments + tracking (420)

jubarte60.24

comments, clean (117)

jubarte68.56

clean (1,188)

jubarte88.11
soffice69.81
PyMuPDF Pro67.45
docxide-pdf57.57

tracking, no comments (1,366)

jubarte82.44
soffice68.39
PyMuPDF Pro57.81
docxide-pdf53.88

comments + tracking (69)

jubarte72.84
soffice46.56
PyMuPDF Pro46.51
docxide-pdf47.24

comments, clean (3)

jubarte69.88
soffice40.69
PyMuPDF Pro41.46
docxide-pdf40.71

clean (1,188)

jubarte79.24
soffice26.33
PyMuPDF Pro31.69
docxide-pdf15.60

tracking, no comments (1,366)

jubarte70.50
soffice22.90
PyMuPDF Pro18.55
docxide-pdf9.95

comments + tracking (69)

jubarte41.71
soffice1.17
PyMuPDF Pro0.26
docxide-pdf0.89

comments, clean (3)

jubarte33.91
soffice4.02
PyMuPDF Pro3.75
docxide-pdf4.51

Oracle: real Word

Every reference PDF is exported by Microsoft Word and SHA-pinned. Redlines are opened in Word too: if Word refuses the file, the tool scores 0 for that pair.

Intent-to-treat

Failures are not dropped. A crash, a timeout or an unopenable file counts as 0 in ITT mean and median, so a tool cannot improve its rank by skipping hard documents.

Paired bootstrap

Ranks tie (n=) when the paired bootstrap interval of the median difference to the row above includes 0. 2,000 resamples, seed 42.

Release samples

From 0.17.0 the tables on this page are the release's two samples — state-balanced, every file sha256-listed in the engine's release_info/. Full-corpus runs remain in the bench's RESULTS.md.

Provenance

where every number came from