AI
Vals aims to overhaul AI model testing with domain-specific benchmarks, raises $40M from A16z
The startup, founded by 25-year-old Rayan Krishnan, is tackling the industry problem of outdated benchmarking systems that fail to assess modern AI capabilities in real-world tasks.
TechCrunch — AI
The pace of AI advancement has left traditional benchmarking methods in the dust. Established metrics were designed for older models and don't adequately test what today's systems can actually do—a gap that companies have learned to exploit, often by training directly on publicly available test data to artificially boost their scores.
Vals, a startup that launched in 2024, is attempting to build a better solution. The company just secured $40 million in Series A funding led by Andreessen Horowitz, following an earlier seed round from 8VC and Bloomberg Beta. Co-founder Rayan Krishnan, now 25, began thinking about this problem while at Stanford, where he worked with Microsoft and the university's AI lab, and later interned at Palantir.
Domain-specific testing, not memorization#
Unlike older benchmarks that measure abstract knowledge through test-like questions, Vals evaluates models on their capacity to execute specialized work. The startup tests AI systems on their ability to perform domain-relevant tasks in law, finance, and software engineering—evaluating whether they can produce outputs that match human-quality work in those fields.
A critical difference: Vals does not publicly release its specific test materials. This prevents companies from gaming results by training their models against known test cases. The startup also goes beyond measuring positive outcomes, examining potential risks by analyzing what problems might emerge if models "ran wild in the world," as Krishnan puts it.
Expanding into emerging territories#
The company's benchmark coverage continues to grow, now touching areas like recursive self-improvement, mental health, cybersecurity, biosecurity, and the law of armed conflict. Vals recently launched a program providing evaluations to federal agencies.
A revenue model that mirrors higher education#
Companies pay Vals to evaluate their models—an arrangement Krishnan compares to the SAT testing model. While it might seem counterintuitive to pay for a test that could reveal weaknesses, the diagnostics help teams identify and fix deficiencies. As more AI acquisitions hinge on independent evaluation results, these benchmarks have become genuine business assets for model developers.
