New Framework Boosts LLM Editing Efficiency by Compressing Context Into Special Tokens

Updating the knowledge baked into large language models has proven tricky: while existing techniques can successfully inject specific facts, they often fall short when faced with tasks that demand deeper reasoning beyond simple fact recall. A new approach called InComeS aims to solve this by giving LLMs a more flexible way to process editing instructions through the model's own contextual reasoning strengths.

The Core Problem#

Traditional model editing methods excel at inserting isolated facts but struggle with semantic understanding and break down as the number of edits increases. Meanwhile, in-context learning—teaching an LLM by embedding edit instructions directly in its input context—shows promise for handling complex reasoning, but runs into a hard wall: the finite context window that constrains how much information a model can process at once.

How InComeS Works#

The framework introduces two key mechanisms to overcome this bottleneck:

• Compression via gist tokens: Each editing context gets compressed into the key-value cache of a special "gist" token, allowing the model to store multiple edits compactly without consuming precious context space.
• Adaptive selection: Custom cross-attention modules dynamically identify and prioritize the most relevant edit information from a pool of gist tokens, ensuring the model uses what it needs without wasteful processing.

This design lets LLMs handle numerous edits simultaneously while staying well within their context window constraints.

Testing and Results#

Experiments across multiple model editing benchmarks and editing formats demonstrate that InComeS improves both effectiveness and efficiency compared to existing methods. The approach appears robust across diverse evaluation scenarios, suggesting it may offer a practical path toward more scalable knowledge updates for large language models.