Fast, Influence-Based Method Removes Unwanted Behaviors from Large Language Models

Removing specific behaviors or knowledge from large language models after deployment remains a significant challenge. Full model retraining is prohibitively expensive, while existing unlearning methods struggle to eliminate targeted problems while preserving the model's useful capabilities—especially when oversight resources are scarce.

The Problem#

Practical machine unlearning for LLMs operates under real-world constraints: organizations work with a small set of examples they want the model to forget, limited data to protect model quality, and the ability to update only a portion of the model rather than retraining everything. Current approaches like Fisher, gradient ascent, and LoReUn either lose too much general utility when erasing targeted behavior or still require substantial computational resources.

How RapidUn Works#

Researchers propose RapidUn, a framework that leverages influence estimates—measures of how much individual training samples impact model predictions—to guide the unlearning process. The method converts these cross-sample influence scores into fixed weights applied during parameter-efficient fine-tuning with LoRA (Low-Rank Adaptation). Rather than computationally expensive iterative updates, this approach applies weighted adjustments directly to a subset of model parameters.

Measured Results#

Testing on Llama-3-8B and Mistral-7B across standard datasets (Dolly-15k and Alpaca-57k), RapidUn achieved better performance than competing methods:

• Lower attack success rates for both seen and out-of-distribution trigger examples compared to Fisher, GA, and LoReUn
• Maintained competitive performance on general language understanding tasks
• On Llama-3-8B with Alpaca-57k, delivered a 77x speedup in wall-clock time compared to standard LoRA retraining on clean data

Additional evaluations using TOFU benchmarks, language model-based semantic assessments, and instruction-following tests (IFEval) provided independent confirmation that influence-guided reweighting effectively removes targeted behaviors across different evaluation approaches.

Why It Matters#

The efficiency gains and maintained model utility make this approach practical for real-world deployment scenarios where organizations need to address problematic model outputs quickly without degrading overall performance or requiring expensive full retraining cycles.