Withdrawn Study Proposed Vector Quantization to Slash LLM Output Layer Parameters

A paper submitted to arXiv proposed a technique called VQ-Logits aimed at addressing a key computational bottleneck in large language models: the massive output layer that converts internal representations into vocabulary predictions. The work has since been withdrawn from the platform.

The Problem: Output Layer Overhead#

Large language models typically use a final linear projection layer that maps hidden states to logits across the entire vocabulary—often hundreds of thousands of tokens. This operation consumes significant parameters and computing resources during inference, since the projection matrix grows directly with vocabulary size. Existing solutions like adaptive or hierarchical softmax methods add architectural complexity.

The VQ-Logits Approach#

The proposed method replaces the full vocabulary embedding matrix with vector quantization, a compression technique that uses a small shared codebook of K vectors instead of V (vocabulary size) unique vectors. Each token maps to one of these K codebook entries, allowing the model to predict logits over this compact space before efficiently expanding back to the full vocabulary.

The technique required learning or assigning a mapping between tokens and codebook vectors, with the model then predicting over the reduced set during inference.

Reported Performance Trade-offs#

According to the paper's experiments on standard benchmarks like WikiText-103 and C4, the approach achieved:

• Up to 99% parameter reduction in the output layer
• 6x speedup in logit computation
• Only a 4% increase in perplexity compared to standard softmax

The authors also provided ablation studies examining codebook size, initialization strategies, and learning approaches to validate the technique's robustness.

Status#

The paper was initially submitted in May 2025 and has since been withdrawn by its author, Jintian Shao. No PDF is currently available on arXiv.