AI
Researchers adapt LLMs to generate music from text, matching human composers in real-world tests
A new approach called MIDI-LLM combines large language models with music generation to produce better text-to-MIDI output and showed strong performance in user studies with musicians.
A team of researchers has developed MIDI-LLM, a method for transforming text descriptions into playable music by teaching large language models to understand MIDI tokens alongside written words.
How it works#
The approach adapts a lightweight version of Llama 3.2 (1B parameters) to handle both language and music data. The team employed a two-phase training strategy: first, they continued pretraining on music-related text and standalone MIDI files to teach the model the fundamentals of musical structure. In a second phase, they fine-tuned the model on pairs of text descriptions matched with their corresponding MIDI compositions.
The resulting system can generate complete multitrack arrangements from text prompts. The researchers also adapted their model to work with lead sheets—simplified musical notation showing melody and chord progressions—by training it further on the TheoryTab dataset for melody and chord generation.
Outperforming existing approaches#
When tested directly against Text2midi, a recent competing system, MIDI-LLM achieved better results in both musical quality and responsiveness to textual instructions. One practical advantage: the model integrates cleanly with optimized inference software like vLLM, making it easier to deploy at scale.
Real-world validation#
The developers conducted a blind user study with 58 musicians using Hookpad Aria, a professional music creation tool. These participants generated over 4,000 musical pieces using MIDI-LLM and baseline systems in actual songwriting sessions. The results showed MIDI-LLM earned the highest acceptance rate for creating new lead sheets from scratch, validating its usefulness in genuine human-AI music collaboration workflows.
An ablation study confirmed that each component—language model pretraining, standalone music pretraining, and supervised fine-tuning—contributed meaningfully to the final performance, suggesting a synergistic relationship among the three.
