French startup Kog targets GPU inference bottleneck with deep-level software optimization

Amid growing industry focus on speeding up AI inference, a Paris-based startup is taking a different tack from purpose-built chip makers. Rather than designing custom silicon, Kog is betting that conventional GPUs—the kind enterprises already deploy in data centers—contain far more untapped potential than current software can access.

The company made waves in May when its tech preview demonstrated that datacenter-grade processors like AMD's MI300X and NVIDIA's H200 could deliver "extremely fast single-request decoding." That proof-of-concept attracted substantial commercial interest: CEO Gaël Delalleau reported over 200 qualified business leads in the months since.

Why speed matters in the AI market#

Inference latency has become a critical economic constraint. Users of advanced coding tools frequently face multi-hour waits for results, while vendors like Anthropic now charge a premium for faster response modes. Companies building AI-powered game and app generators see direct revenue implications in reduced output time—every second of delay represents lost customer productivity.

The problem is that current software doesn't make full use of what modern GPUs can deliver. Delalleau contends that the perception of GPUs as poorly suited for language model inference is a misconception. Newer accelerators pack increasingly generous memory bandwidth; the challenge is a software one.

A physics-first approach rooted in hacking#

Delalleau's background shaped Kog's methodology. He holds a degree in solid-state physics and spent years in offensive security, competing four times in DEF CON's CTF tournament. That dual training instilled in him a discipline: combining rigorous physics principles with low-level reverse-engineering to understand—and exploit—hardware behavior at assembly and binary levels.

This mindset now drives Kog's team of 11. For each GPU architecture, the company commits weeks or months to detailed engineering research, extracting maximum performance through intimate knowledge of the underlying silicon. The approach is labor-intensive but yields results that general-purpose optimizations cannot match.

Scaling the proof to large models#

Kog's initial demo achieved an impressive using a small custom model (Laneformer 2B, now ) with roughly 2 billion parameters. The critical next step is replicating that acceleration on full-scale large language models—a substantially harder engineering problem.