Training Smarter AI Models by Mining Your Own Production Data

Developer tools infused with AI often require expensive API calls and generic training that doesn't match individual workflows. One alternative: harvest the richest signal source already at hand—your own system telemetry—and use it to distill a custom, efficient model.

The Problem: Expensive, Generic AI in Developer Tools#

Building an AI-powered Language Server Protocol (LSP) for code editing felt compelling in theory: semantic analysis, intelligent suggestions, proactive insights woven into the editor. But the reality proved costly. Every save triggered API calls to frontier models, burning tokens throughout the day. More fundamentally, there was no way to customize the "brain" of the tool or improve it permanently over time. Swapping models offered marginal gains, but true personalization remained out of reach.

The underlying tension: frontier models solve the hard problem of semantic understanding, but they can't be tuned to individual habits, codebases, or preferences without rebuilding from scratch.

Capturing Behavioral Feedback Through Observability#

The turning point came from recognizing that OpenTelemetry, already capturing every interaction, was generating the exact training signal needed. Each accepted suggestion, rejected fix, and regeneration request became labeled data—without any extra labeling effort.

By instrumenting the LSP with OpenTelemetry spans, three behavioral signals emerged naturally:

• Accepted (strong positive): user applies a suggested fix, validating both problem detection and proposed solution.
• Regenerated (weak positive): user agrees with the diagnosis but asks for an alternative, signaling "on the right track, try again."
• Dismissed (strong negative): user ignores a suggestion entirely—the rarest and most valuable signal in traditional ML.

Unlike typical thumbs-up-or-down feedback, these signals reflect actual usage patterns, not what users think they think. Over a month of normal coding, dozens of interactions accumulate; within weeks, hundreds of labeled examples emerge without any manual annotation burden.

From Traces to Training Data#

Converting OpenTelemetry logs into a training dataset requires straightforward mechanics: export traces to a backend (Logfire, Honeycomb, or local JSON via OTel Collector), filter for spans containing user actions, then extract positive/negative examples into JSONL format. The heavy lifting was instrumentation; the extraction pipeline itself stays simple because the telemetry already captures the right structure.