Go Team Builds Kafka Pipeline That Preserves Message Order Within Sessions While Enabling Cross-Session Parallelism

Building a scalable messaging system for conversational AI demands that related messages stay in order—when a user changes their flight destination from London to Paris, the system must process that correction second, not first. A team at an enterprise conversational AI platform tackled this at scale by layering application-level ordering logic on top of Kafka.

The Core Problem#

Kafka natively guarantees message ordering within a single partition, but that alone isn't enough. When thousands of unrelated user sessions share the same partitions—the efficient way to operate Kafka at scale—messages from different sessions interleave. The system needed a way to process sessions in parallel without mixing their message sequences.

Partition-per-session doesn't work: that would require thousands of partitions, exceeding operational limits and harming broker performance. A single-threaded model per partition is wasteful when sessions are multiplexed. The solution required application code to route and sequence messages independently of partition boundaries.

Two-Level Worker Architecture#

The production design uses a two-tier worker hierarchy. A pool of lightweight dispatch workers receives records from Kafka and routes them by session ID using consistent hashing, ensuring each session always reaches the same worker. Inside each worker, individual goroutines handle one session at a time, processing messages sequentially.

This design achieves unlimited parallelism across sessions while guaranteeing strict sequencing within each one—by construction, not by luck. Sessions are created on demand and cleaned up after inactivity, keeping resource usage proportional to active chat sessions.

Retries Without Overtaking#

The first attempt used a separate retry goroutine that queued backlogged messages in memory, creating complex state machines and unbounded memory risk. The current approach is elegantly simple: retries happen inside the session goroutine itself.

When a message fails due to a transient error (downstream service timeout, network blip), the goroutine sleeps using exponential backoff with jitter, then retries the same message. Because the goroutine is blocked, subsequent messages queue behind it in the session channel—they literally cannot execute first. Non-recoverable errors skip straight to the dead-letter queue. Once the failed message succeeds or is abandoned, the next message processes immediately.

This eliminates retry state machines entirely. A blocked goroutine and a filled channel are both sufficient to preserve order.