AI
New Framework Helps AI Models Learn Private Company Code Libraries Through Execution Feedback
MEMCoder uses a self-evolving memory system that improves code generation for internal libraries by learning from execution results instead of relying solely on API documentation.
Large language models excel at generating code for common programming tasks, but they struggle significantly when asked to work with proprietary internal libraries that never appeared in their training data. Researchers have now proposed a solution that learns directly from execution feedback rather than static documentation alone.
The Documentation Gap#
Current retrieval-augmented generation (RAG) systems help by providing relevant API documentation to guide code generation. However, this approach has clear limits: knowing which API to call is different from understanding how to use it correctly. Analysis of existing systems shows that even when given perfect API documentation, LLMs still make systematic mistakes at three levels—misusing individual APIs, combining APIs incorrectly, and pursuing flawed overall strategies.
How MEMCoder Works#
Researchers introduced MEMCoder, a framework that builds a multi-level memory system on top of standard RAG pipelines. The system accumulates practical "Usage Guidelines" at three granularity levels: individual APIs, combinations of APIs, and complete task-solving approaches. These guidelines come from analyzing actual execution results rather than abstract documentation.
The process operates in a closed loop:
- During generation, the model retrieves both traditional API documentation and relevant historical usage patterns
- After code executes, the system analyzes what went wrong (or right)
- Feedback is converted into concrete usage principles
- The memory evolves with each iteration, refining guidance for future attempts
Demonstrated Improvements#
Testing on benchmark datasets (NdonnxEval and NumbaEval) showed consistent gains: MEMCoder improved code generation accuracy by an average of 18.41 percentage points across different LLM sizes and RAG backends. The approach also outperformed other self-evolving memory methods, validating the effectiveness of organizing multi-level usage knowledge derived from execution.
Why It Matters#
For enterprises relying on custom libraries and internal APIs, this addresses a critical gap where off-the-shelf LLM capabilities break down. By learning from real execution outcomes rather than documentation alone, the system adapts to the specific patterns and requirements of proprietary code ecosystems.
