New sparse attention method cuts LLM inference time by up to 38% without sacrificing accuracy

Long-context reasoning in large language models comes at a steep computational cost. The quadratic complexity of attention mechanisms has become a major bottleneck for real-world inference, especially as models process increasingly longer documents and conversations.

A new technique called Block Sparse Flash Attention (BSFA) offers a way around this limitation. Researchers have developed a method that preserves the accuracy of modern models while substantially reducing the time and memory overhead of attention computation.

How it works#

Rather than predicting which tokens matter before computing attention scores, BSFA takes a different approach. It calculates exact query-key similarities across all token blocks, then selectively retains only the most important value blocks for each query. By comparing maximum attention scores at the block level against calibrated thresholds, the method can skip roughly half the computation and memory transfers for less important blocks.

The threshold calibration is a one-time process performed on a small calibration dataset. This step learns the typical distribution of attention scores across layers and heads, making the method training-free and lightweight to deploy.

Performance gains#

On Llama-3.1-8B, BSFA achieves speedups of up to 1.13x end-to-end on the LongBench benchmark with minimal accuracy loss (1.1% drop). For retrieval-focused tasks measured by Needle-in-a-Haystack tests, the method reaches 1.24x speedup at a similarly small 1% accuracy cost. The attention kernel itself runs up to 1.38x faster.

The authors implemented BSFA as a CUDA kernel compatible with FlashAttention, allowing it to function as a drop-in replacement. Testing included benchmarks on Qwen2.5-7B and comparisons against five competing sparse attention methods including SpargeAttention, MInference, FlexPrefill, XAttention, and BLASST. The approach was validated on both A6000 and H100 GPUs.

Why it matters#

As language models tackle increasingly complex reasoning and multi-document retrieval tasks, the ability to handle long contexts efficiently becomes critical. This work demonstrates that selective computation—skipping tokens that don't require attention processing—can deliver meaningful speedups without requiring model retraining or substantial accuracy sacrifices. The ease of deployment as a plug-in replacement makes adoption practical for existing model deployments.