Measuring fluency
Juan goes down the rabbit hole of how language level is actually scored, and comes back with a sinsabor: there's no gold standard, only competing approximations. The signals that seem to work: higher levels (C1, C2) show up as complex sentences, less-frequent vocabulary, and abstract 'ethereal' images rather than concrete ones ('the boy fell'). Essay complexity, the fancier your lexicon and grammar, the higher your score, which is exactly how a teacher grades a composition. LLMs turn out to be reliable at scoring text: he had ChatGPT write a C2 passage and ChatGPT, Claude, and Gemini all graded it back around C1-C2. For the other skills he'll use rougher thresholds: listening by words-per-minute (A1 60-80, C2 180+), speaking by response time (A1 3-6 seconds, A2 2-4). He dislikes the hand-tuned thresholds but calls it a start, and dreams of the real fix: a GeoGuessr-style global ranking where users translate the same phrases, a scientific ranking instead of an empirical guess.
// trace: where this idea came from
- ↳ video diary @ 8:00 (no hay una regla de oro para calificar niveles, solo aproximaciones)
- ↳ video diary @ 15:03 (los LLMs son muy buenos calificando texto, ChatGPT, Claude y Gemini coincidieron)
- ↳ video diary @ 15:53 (escucha por palabras por minuto, habla por tiempo de respuesta)
- ↳ Entry 252-1: Two sides of the same coin (la consolidación cuyo umbral aquí se intenta medir)
Having decided the reflex is the second phase, Juan went looking for how language level is actually scored, and came back with a sinsabor: there’s no gold standard, no single indicator everyone agrees on, only competing approximations ▸ 8:00. The signals that seem to work are about complexity. Higher levels show up as complex sentences, less-frequent vocabulary, and abstract images rather than concrete ones; “the boy fell” is a picture you see at once, an Aristotle reference is not ▸ 8:53. It’s the same thing a teacher does grading a composition, the fancier your lexicon and grammar, the higher the mark, which he half-dislikes but concedes works ▸ 11:37.
no hay regla de oro; la fluidez se mide en velocidad y complejidad →
For text, LLMs turn out to be reliable graders: he had ChatGPT write a C2 passage, then fed it back cold to ChatGPT, Claude, and Gemini, and all three scored it around C1-C2 ▸ 15:03. For the other skills he’ll use rougher thresholds, the hand-tuned kind he doesn’t love but calls a start: listening by words per minute (A1 around 60-80, C2 past 180), speaking by response time (A1 three to six seconds, A2 two to four) ▸ 15:53. What he really wants is to stop guessing: a GeoGuessr-style global ranking where users translate the same phrases and sort themselves against each other, a scientific ranking instead of an empirical one ▸ 19:03. Measure the transition, don’t declare it…