New Model Cuts Energy Use 10-Fold by Treating Long Context as a Single Data Type

Researchers have proposed a novel approach to running language models efficiently on edge devices, addressing a persistent challenge in making AI responsive and sustainable in resource-constrained settings. The new architecture, called Dolphin, leverages a creative strategy: compressing long text passages into a single unified representation before processing them.

How Squid Works#

At the heart of Dolphin lies a two-stage decoder design. A smaller, lightweight decoder with 0.5 billion parameters first ingests the full context and distills it into a dense memory embedding—essentially a condensed summary of the salient information. This compressed representation then feeds into the main 7-billion-parameter decoder, which performs the actual language modeling task. By reducing what the larger model must process, the system avoids the computational bottleneck that typically accompanies long-context handling.

The trick is treating this distilled context as a new modality rather than as ordinary text. Drawing inspiration from vision-language models, the researchers repurposed an image embedding projector to encode the textual context, allowing the architecture to handle extended input without the overhead normally required.

Performance Gains#

Empirical testing reveals substantial efficiency improvements. The model achieves a 10-fold reduction in energy consumption and 5-fold decrease in latency relative to standard full-context processing, all while preserving answer quality. For on-device applications—where power budgets and response times are critical—these gains represent a meaningful advance.

Broader Implications#

Dolphin opens a path toward more capable edge AI. By compressing long contexts into a single dense representation, the approach makes it feasible for resource-limited devices to handle sophisticated language understanding tasks that would otherwise require cloud offloading. This matters for privacy-sensitive workloads, low-bandwidth environments, and applications where latency is unacceptable. The model is publicly available for research and practical deployment.