AI
Medical Schools Deploy AI-Powered Grading Platform for Video, Audio, and Text Assessments
OASIS automates rubric-based performance evaluation at scale across multiple content types, combining LLM scoring with human oversight and audit trails.
A new assessment platform called OASIS enables institutions to evaluate student performance across videos, recordings, and written work using large language models while maintaining strict oversight and traceability. The system has been running in production at UT Southwestern Medical Center since fall 2023, handling more than 7,000 assessment encounters.
How It Works#
OASIS combines a command-line tool with an integrated backend stack built on Elephant and MAPLES systems to manage evaluations at scale. The platform applies structured rubrics to graded material and generates per-criterion scores, supporting evidence, and detailed rationales for each assessment. Crucially, the system preserves a complete audit trail of all scoring decisions and intermediate outputs.
The architecture supports flexible model deployment—evaluators can point OASIS to hosted LLM APIs or run self-hosted open-weight models through Ollama and vLLM-compatible endpoints. The platform treats rubrics as executable programs that compile into progressive evaluation plans, making scoring logic transparent and reproducible.
Key Features#
OASIS distinguishes itself through several technical innovations:
- Multimodal grading: Transcripts enhance video and audio evaluation, allowing the LLM to factor in spoken content alongside visual cues
- Content-addressable identity: Grading decisions are cryptographically linked to inputs, preventing mismatched scores or lost context
- Explicit review state: Human reviewers have clear visibility into which assessments are pending, approved, or flagged for revision
- Unified command surface: The same interface serves both human operators and autonomous agents, eliminating manual-to-automated handoffs
Optional extensions include SimRubrics for rubric design and Wayfinder, a conversational gateway that helps evaluators interact with the system more naturally.
Domain-Agnostic Design#
Although developed for medical education, the architecture applies wherever recorded performances or written work can be systematically evaluated against defined criteria. The platform's emphasis on encounter management, rubric versioning, and human review makes it applicable across professional licensing, competency assessment, and quality assurance contexts.
The publication shares the system design and operational results but does not include source code, binaries, or sample datasets.
