Essays - Benedict Evans · Benedict Evans
articleinsightAre better models better?
22 January 2025Strategy
Source excerpt
About Essays - Benedict EvansEvery week there’s a better AI model that gives better answers. But a lot of questions don’t have better answers, only ‘right’ answers, and these models can’t do that. So what does ‘better’ mean, how do we manage these things, and should we change what we expect from computers?
This is a limited feed-provided excerpt, not the full original work.
Apollo layer
What a founder can learn
Founder takeaway
Define AI quality by the job being done, not by general model capability. Separate tasks where incremental improvement is valuable from tasks that require a verifiably correct answer, then design evaluation and human oversight accordingly.
Why it matters
A broadly better model may still be unsuitable for workflows where one wrong answer creates outsized risk. Task-specific standards help founders set honest expectations, choose models rationally, and build safeguards where correctness matters.
Relevant guides
Put it to work
List your product’s top AI-assisted tasks. For each, ask: Is a plausible improvement useful, or must the output be correct? What test, constraint, or review step matches that standard?