Benchmarks for humans
How do you know you're doing well? Society answers with grades, family scripts, job titles, and when you step outside every system, the last indicator standing is money, 'lo mejorcito entre las cosas que hay'. LLM benchmarks are the same machinery: invented tests that become the forces pulling the field, made by the very companies competing on them.
// trace: where this idea came from
- ↳ video diary @ 1:07 (¿cuáles son los indicadores de que va bien?)
- ↳ video diary @ 4:12 (los benchmarks como guías inventadas)
- ↳ Entry 162-1: From 69 to 95 (la investigación que busca la regla única)
The question sounds simple: “¿cómo sabe uno que está haciendo las cosas bien? ¿cuáles son los indicadores?” ▸ 1:07. Society answers in layers: school grades ▸ 1:43, the family script of stable job, house, car ▸ 2:00. The founder’s problem is stepping outside every one of those systems at once ▸ 2:30, and his reluctant conclusion about what remains: “el indicador principal, aunque no lo quiera aceptar, termina siendo el dinero” ▸ 2:40, not because it’s good, but because it’s “lo mejorcito de entre las cosas que hay” ▸ 3:03.
Then the isomorphism that titles the video. LLM launches ride on benchmark charts ▸ 3:38, and those tests are exactly grade-school report cards for machines: invented by us, guiding the models and telling the labs they’re doing well ▸ 4:12, with the twist that the companies being graded also author the tests, then compete on them, forces pulling the whole field in whatever direction the metric points ▸ 4:44. Products live under the same law: KPIs, conversions, time-on-page are self-imposed benchmarks ▸ 5:20.
quien elige la métrica elige la dirección →
His research question sits inside the same frame, inherited from the loss-as-reward thesis: reinforcement learning works through piles of imposed rules that leave the system complex, so could one simple rule suffice ▸ 6:31? And the practical corollary for humans is metric hygiene: quantify what actually improves the thing, timing your runs helps, tracking muscle mass doesn’t ▸ 8:07. Even mathematics plays: axioms are rules chosen because they cohere with lived experience ▸ 9:16, which leaves his best joke as an open research problem: how does the axiom-maker know if he’s doing well or just “fumando mucho” ▸ 10:02? Background listening for all this: five hours of Lex Fridman with Anthropic, Dario Amodei and the philosopher he describes as having given Claude its personality ▸ 12:51…