Two minds, one model
A true/false bug exposes a subtle failure of stateless LLM calls. Severo generates an exercise with a deliberate error, the user answers 'false' correctly, but the evaluation call marks it wrong, because it's a separate call than the generation, so 'they're like two different people thinking'. The one that wrote the error and the one that grades it don't share a mind, and the grader can reason its own way to the wrong verdict. Juan's fix: have the generating call write down its reasoning (why this is the error) somewhere the grading call can read, so the evaluation judges against the intent instead of re-deriving from scratch. The general shape: when two calls must agree, don't make them think twice, make the first one leave a note for the second.
// trace: where this idea came from
- ↳ video diary @ 10:42 (entre una llamada y otra son dos personas pensando distinto)
- ↳ video diary @ 11:03 (que el que genera deje el razonamiento para el que evalúa)
- ↳ Entry 229-1: The lemma is the unit (el mismo motor cuyos errores aquí se persiguen)
A true/false exercise surfaces a subtle bug. Severo, going English to Portuguese, generates a sentence with a deliberate error, “o sofá é na sala” ▸ 9:28. Juan answers false, correctly, and Severo insists the correction is a different one and marks him wrong. The model, in short, hallucinates about its own exercise ▸ 9:35.
The diagnosis is what makes it a teaching. The exercise is generated in one API call and evaluated in another, and “entre una llamada y otra son como dos personas pensando diferente” ▸ 10:42. The call that planted the error knows it’s an error; the call that grades, though it’s the same model, is a fresh mind with no memory of the intent, free to reason its own way to “actually, that’s fine” ▸ 9:58. Statelessness, the feature that lets each call start clean, is exactly the flaw here: two invocations of one model do not share a thought.
dos llamadas no comparten mente; deja una nota →
The fix is to stop asking two minds to independently reach the same conclusion. Have the generating call write down its reasoning, why this sentence is the error, somewhere the grading call can read it ▸ 11:03, so the evaluation judges against the stated intent instead of re-deriving the answer from a blank page. It’s the same engine whose confident errors the whole backend keeps having to police, and the general shape generalizes past this bug: when two calls must agree, don’t make the second one think twice, make the first one leave a note. A model that forgets between calls will contradict itself unless you hand it back the memory it dropped…