2023: The Year Open Large Language Models Took Off

How Open LLMs Work#

A large language model combines three key ingredients: model architecture (typically a decoder-only Transformer), a training dataset (often hundreds of billions to trillions of tokens), and a tokenizer (which converts text into numerical sequences). Training hyperparameters then govern how the model learns from this data. The result is a set of weights that can be fine-tuned for specific tasks—a process far cheaper than training from scratch, which made 2023's open-source releases so valuable.

The Chinchilla Effect#

In early 2022, the prevailing wisdom was that bigger meant better. Models like BLOOM (176B parameters), Meta's OPT (175B), and Tsinghua's GLM-130B dominated the landscape.

That changed when DeepMind published research showing the optimal ratio of tokens to parameters for a given compute budget. Their finding: smaller models trained on more data performed as well or better than enormous ones. This insight sent shockwaves through the open-source community.

2023: The Deluge of Releases#

The result was a tidal wave of releases. LLaMA (Meta, February), StableLM (StabilityAI, April), Falcon (TIIUAE, June), Llama 2 (Meta, July), Mistral (Mistral AI, September), and dozens more followed—most in the 3B–70B parameter range, trained on substantially more data.

Key trends in these models:

• Efficient architectures with tweaks like ALiBi, RoPE, Flash-Attention, and grouped query attention
• Weights released under varying open licenses, enabling immediate community adoption
• Strong performance per compute budget—the 13B LLaMA model outperformed GPT-3
• Most came with both a base pretrained version and a chat-finetuned variant

Notable late-year releases included Alibaba's Qwen (7–70B, trained on 2.4T tokens) and 01-AI's Yi (6–34B, 3T tokens), which demonstrated that models trained outside the Western AI ecosystem could match or exceed peers on open benchmarks.

Dialog Models Everywhere