New benchmark helps developers pick the right document compilation engine

Choosing the right document compilation engine for LaTeX and related workflows has lacked a principled, evidence-based approach. A new study from arXiv addresses this gap with Engine-Transfer-Bench (ETB), a comprehensive testing framework that evaluates six major document compilation tools across reliability, speed, and output consistency.

What is Engine-Transfer-Bench?#

ETB comprises 1,784 real-world documents tested against four evaluation criteria: compilation success rates, execution time, text consistency in PDF output, and failure patterns. The framework includes a standardized testing harness pinned to specific versions and runs on GitHub Actions across macOS, Ubuntu, and Windows, generating over 4,200 compilation results per system. Researchers also developed ETB-Porta, a recommendation tool and portability check to help users select appropriate engines.

Key findings on reliability and performance#

The engines tested—pdfLaTeX, XeLaTeX, LuaLaTeX, Tectonic, Typst, and pandoc PDF backends—show dramatically different behavior across operating systems. Tectonic proves notably consistent, succeeding 96.3–97.2% of the time regardless of platform, a variance of less than one percentage point. By contrast, traditional TeX Live-based engines fluctuate by 12–20 percentage points depending on how their packages are distributed on each system (Ubuntu apt, MiKTeX auto-install, or macOS BasicTeX).

On a narrower set of 702 portable LaTeX documents, all tested engines achieve perfect success, suggesting that latency becomes the primary deciding factor for well-authored, self-contained documents. The authors identify that failures are concentrated within 107 engine-specific templates and stem from architectural limitations—font availability, layout handling, and asset management—rather than missing packages on a provisioned system.

Text consistency and validation#

The team validated a text-consistency metric called S_pdf by examining 50 document pairs. The metric achieves 94% precision in detecting genuine differences in PDF content, providing a reliable way to assess whether engine choice affects the actual rendered output users see.

Implications for the community#

By releasing ETB along with open infrastructure and tools, the researchers create a shared reference point for making engine selection decisions grounded in data rather than convention or trial-and-error. This standardization may help reduce compatibility headaches in collaborative and production document workflows.