New system learns from failures to improve answers about corporate financial filings

Researchers have developed a novel approach to making financial question-answering systems more reliable by learning from their mistakes over time. The system, called FINSKILLOPS, tackles a persistent challenge in deploying AI for SEC filing analysis: how to improve performance after launch without introducing new errors.

The problem with standard improvement#

Traditional financial QA systems are typically refined before release through better data retrieval, prompt engineering, or multi-agent coordination. Once deployed, their behavior remains static. In practice, new questions regularly expose different kinds of errors — miscalculations, incorrect entities, wrong time periods, or misuse of evidence — that a fixed system cannot address.

Existing self-improvement methods can turn failures into corrections, but they lack precision: changes intended to fix one problem may inadvertently break answers that were already working correctly. This unpredictability makes them risky for financial systems where accuracy is critical.

How FINSKILLOPS works#

The framework reframes post-deployment improvement as controlled behavioral maintenance. Rather than making broad adjustments, the system:

• Diagnoses failures with precision — identifying the specific type of error (period, entity, evidence, calculation) using typed, evidence-grounded analysis
• Creates targeted skill patches — transforming each diagnosis into a reusable correction that addresses just that category of error
• Validates before deployment — each new patch must pass targeted checks, regression tests on previously correct answers, negative controls, and other safeguards
• Manages versions — patches can be refined or retired, keeping a versioned registry of approved skills

Results and evaluation#

Testing across six financial benchmarks, the frozen skill registry achieved the best overall correctness and consistency among compared systems. On an enhanced benchmark, evolved skills improved correctness from 3.70 to 4.55.

In a 12-round operational study simulating real-world deployment, only six of 33 proposed skills were promoted to production. Despite this selective approach, the error rate dropped from 20.0% to 12.5%, suggesting that strict admission criteria prevent low-quality patches from harming the system.