New Training Method Lets AI Music Models Learn Whole Compositions at Once

Researchers have developed a practical approach to train symbolic music language models on complete compositions, addressing a fundamental limitation in how these systems currently learn musical structure.

The Problem with Fragmented Training#

Most language models today are trained on fixed-length text sequences sampled independently for computational efficiency. Music models have adopted the same approach, splitting compositions into isolated chunks. This fragmentation breaks the model's ability to learn from and condition on the musical context that unfolds across an entire work—a natural way music actually develops structurally.

Full-Horizon Compressed Recurrence#

The team's solution, called Full-Horizon Compressed Recurrence (FHCR), preserves access to an entire composition's history during training while keeping memory demands practical on standard GPUs. Rather than truncating how far back the model can "look," FHCR compresses the internal memory representation—the key-value data structures that recurrent attention mechanisms rely on—making it feasible to train on whole pieces without prohibitive memory overhead.

Measuring Long-Range Learning#

To validate that models actually use this extended context, the researchers introduced a diagnostic called KV-Reset Context Utilization (KRCU). Testing on the MAESTRO symbolic piano dataset, KRCU revealed that full-horizon models meaningfully draw on context far beyond local segment windows. By contrast, models with truncated memory horizons showed substantially weaker long-range dependency—indicating they truly lose access to distant musical information.

Key Insight#

The findings underscore that preserving temporal extent matters more than reducing recurrent memory size by shortening the horizon. Compression of internal representations, rather than forgetting older context, proves a more effective path to practical whole-piece training.