New benchmark exposes gaps in discrete visual tokenizers for image generation

Autoregressive image generation models have emerged as a powerful approach in recent years, but their performance hinges on an often-overlooked component: the visual tokenizer that converts continuous pixel data into discrete token sequences. Researchers have now created the first comprehensive benchmark to evaluate how well these tokenizers preserve image quality—and the results reveal significant room for improvement.

The bottleneck in token-based generation#

Visual tokenizers act as a gatekeeper for autoregressive models, establishing the upper limit of what these systems can produce. Current discrete implementations fall well short of their continuous counterparts, resulting in reconstructions that lose fine-grained detail, texture fidelity, and text integrity. Yet most existing evaluation frameworks focus on end-to-end generation quality, making it difficult to isolate where tokenization failures occur.

Introducing VTBench#

A new research paper introduces VTBench, a systematic evaluation framework designed specifically to measure visual tokenizer performance across three key dimensions:

• Image reconstruction: How accurately the tokenizer captures the original visual signal
• Detail preservation: Retention of fine-grained textures and spatial structure
• Text preservation: Ability to maintain legible text within images

The benchmark covers diverse evaluation scenarios and applies a range of metrics to assess reconstruction quality.

Key findings#

The research confirms a clear performance gap between approaches. Continuous variational autoencoders (VAEs) produce markedly superior visual representations, maintaining spatial coherence and semantic integrity. By contrast, discrete tokenizers frequently generate distorted reconstructions, lose textural subtlety, and struggle to preserve both text and object boundaries intact.

The team also examined potential autoregressive properties of GPT-4o's image generation, offering fresh perspective on how tokenization shapes model behavior in production systems.

Looking forward#

To accelerate progress, the researchers have released both the benchmark and supporting code openly, calling on the community to develop stronger, production-ready open-source visual tokenizers that can better match the performance of continuous alternatives.