Autograders Made Faster and More Accurate by Starting With Solution Code

Creating reliable autograders for programming assignments has long been a bottleneck for educators. Instructors typically must write numerous test cases by hand to ensure their graders correctly evaluate student submissions—a time-consuming process prone to gaps and errors.

Researchers propose a faster alternative: solution-generated autograding, which flips the workflow by starting with the instructor's own reference solution rather than building test cases from scratch. This shifts the focus from enumerating exhaustive test cases to deriving test coverage directly from working code.

How It Works#

The key insight is that instructors authoring an assignment already have a complete, correct solution in hand. Rather than force them to independently specify all edge cases and test conditions, the solution-generated approach automatically extracts meaningful test scenarios from that solution. This eliminates manual test enumeration, reduces the need for post-hoc validation of the autograder itself, and enables instructors to assess submission quality beyond just whether code produces the right output.

Questioner: A Practical Implementation#

Researchers developed Questioner, a tool implementing this method for Java and Kotlin programming assignments. The team piloted the approach at scale over four years in a large introductory CS course, where it powered the authoring of nearly 800 programming problems. Those assignments collectively evaluated millions of student submissions, demonstrating both feasibility and impact.

Why It Matters#

When autograder creation becomes less burdensome, instructors can build larger problem banks to support beginning programmers—providing more opportunities for practice and feedback. The approach also shifts quality assurance earlier in the workflow: by grounding test generation in working code, instructors gain confidence in correctness while spending less effort on manual case enumeration.