AI
Arm & ExecuTorch 0.7 bring generative AI to older smartphones and edge devices
With optimized Int4 matrix multiplication leveraging the SDOT CPU instruction, models like Llama 3.2 can now run efficiently on devices up to five years old and even Raspberry Pi.
Hugging Face Blog
Arm and the ExecuTorch team are democratizing on-device generative AI by enabling large language models to run on older and lower-end hardware. The upcoming ExecuTorch 0.7 beta will ship with Arm's KleidiAI—an AI accelerator layer—enabled by default, bringing automatic performance optimizations to Android devices without requiring code changes from developers.
Making GenAI practical on older devices#
The key enabler is the SDOT (Signed Dot Product) instruction, an Arm CPU feature available since 2015. This instruction efficiently handles dot product operations on 8-bit signed integers—the core computational workload for LLMs. Approximately 3 billion Arm-based devices already support SDOT, representing 72% of all devices worldwide.
This means generative AI experiences that were once limited to flagship phones can now run on devices 3–5 years old, as well as edge devices like the Raspberry Pi 5. When Arm and ExecuTorch teams optimized Llama 3.2 1B with Int4 matrix multiplication, they achieved:
- Over 20% higher prefill performance on Galaxy S24+ compared to non-KleidiAI kernels
- 350+ tokens per second during prefill
- 40+ tokens per second during decode
- Sufficient performance for on-device tasks like summarizing unread messages with smooth user experience
Real-world use cases#
Even with SDOT-only optimization on older Arm CPUs, practical applications emerge:
- Private smart assistants: Combined with speech-to-text and text-to-speech models, fully offline voice assistants require no cloud connection or data sharing
- Context-aware text completion: Real-time suggestions in local text editors as you type, without internet access
- Local inference: Decode speeds often exceed average human reading speed, making inference feel responsive
The optimization works by embedding KleidiAI into widely-used frameworks including XNNPack, MediaPipe, MNN, ONNX Runtime, and llama.cpp—each gaining automatic acceleration with no developer integration required.
Arm is providing a learning path for developers ready to build Android apps with Llama, KleidiAI, ExecuTorch, and XNNPACK.
