Oral assessment with generative AI requires more than a conversational interface. If a model records, transcribes, questions or scores a learner, educators need to be able to connect those operations back to the source material, the rubric, the model outputs and the final human judgement.
My new technical report, “CALLIOPE: A Source-Grounded Oral Assessment System and Synthetic Readiness Evaluation,” documents an implemented response-to-review workflow built around that requirement.
CALLIOPE combines versioned instructional material, learner-turn recording and transcription, adaptive questioning, two-provider rubric scoring, educator review and exportable evidence. The aim is not to automate away educational judgement. It is to make the route from a spoken response to a reviewable decision more inspectable.
What the current evidence shows
The report analyses retained synthetic verification records from 25 September 2026. Three spoken fixtures and a silence control were exercised across two release runs. The final rehearsal produced differentiated aggregate scores, surfaced provider-disagreement flags in two fixtures and recovered audio files that were byte-identical to their inputs.
These observations establish only that specific software paths operated as recorded. They do not establish that the scores are valid, that the system improves learning, that it is usable across populations or that it is ready for unsupervised deployment. No human-participant outcomes are reported.
That distinction matters. Research software should make both its evidence and its limits visible. The next stage should involve carefully governed human-participant work, explicit assessment-validity questions and close educator involvement.
I welcome conversations with assessment researchers, educators and institutions interested in evaluating source-grounded oral-assessment workflows while keeping people responsible for consequential judgements.
Read the technical report: arXiv · SSRN · DOI
Publication note: this is a technical report/preprint. Availability on arXiv and SSRN is not peer-review acceptance.
Leave a Reply