Gavin Reid

2026 · Independent project

CEFR speech scoring lab

ASR, fluency features, agreement and fairness evaluation for L2 speaking assessment.

Architecture of CEFR speech scoring labASR, fluency features, agreement and fairness evaluation for L2 speaking assessment. licensed corpus 876 responses 438 speakers human scores gold STMs whisper-small word timestamps Praat prosody from raw audio 17 features fluency · prosody scorer ladder grouped 5-fold evaluation QWK · r · MAE band calibration WER · partial r fairness slices human scores · gold STMs

What it is

An automatic speaking-assessment scorer built so that the evaluation around it can be written up properly. The scorer is the means; the report is the product. It is not trying to be state of the art, and saying so is the whole point.

It runs on a licensed research corpus of second-language English speech: 876 long-turn responses from 438 speakers, each with a single consensus proficiency label on a half-point scale.

The problem

I taught English for years, so I know what an examiner is actually doing when they put a number on someone’s speech. That is a judgement with consequences, and the interesting question about automating it is not “how accurate” but “accurate for whom, how consistently, and where does it fail”.

Most speaking-scorer write-ups answer the first question and skip the rest. The gap I wanted to close is the evaluation science: agreement, error structure, subgroup behaviour, and the honest statement of what the data cannot tell you.

How it works

Whisper transcribes each response with word timestamps. Eleven fluency features are computed from those transcripts: speech rate, articulation rate, pause counts and durations, long pauses, mean length of run, filled pauses, and lexical diversity. Five prosody features come from the raw audio instead, so they do not inherit the transcriber’s errors. Pitch spread is reported in semitones, and intensity means are dropped entirely because they say more about the microphone than the speaker.

The lexical-diversity slot is the one I would point at. Type-token ratio correlated negatively with proficiency, which is the textbook length-sensitivity artefact rather than a finding about learners, so it was replaced with MTLD, which correlates positively and lifted the model.

Three estimators are compared against two deliberately stupid baselines, cross-validated with folds grouped by speaker, because the same speaker contributes two responses. Switching to random folds measured that hazard instead of assuming it: Ridge gained 0.015 quadratic weighted kappa and the random forest gained nothing at all, which is itself evidence that these features barely encode speaker identity beyond proficiency. Random forest wins at 0.558, against 0.500 for a baseline given nothing but word count and which part of the test it was. Agreement is 34.1% exact, 81.5% within half a band, 97.7% within one band.

The word-error-rate analysis is the part I am most pleased with. Transcription accuracy correlates with score error, which looks like an ASR problem, until you control for proficiency and the correlation vanishes. It was proficiency in disguise.

What I’d change

The corpus ships one consensus label per response, so inter-rater reliability cannot be computed at all and there is no human ceiling to compare the model against. It carries no first-language metadata either, which means the most important fairness slice in the assessment literature is unavailable. Both are stated limits, not oversights, but they cap what the report can claim.

The model also compresses the scale, flattering the weakest candidates and penalising the strongest. That alone makes it unfit for any decision about a real candidate.