The film, in writing.
A model can make one job faster and easier, then give a plausible but wrong answer to the next. I start with a weekend to-do list to make that pattern easier to see. Fluent writing doesn’t tell us which answers we can rely on.
The model’s capabilities are one frontier. The useful work we can do within limits of quality, time and cost is another. A higher benchmark score can be encouraging without telling a team what will improve in its own work.
In the BCG consultant experiment, the tool helped on tasks within its capabilities and reduced correctness on a task outside them. I’d use that result as a reason to check actual tasks with people who know the work. Joshua Gans’s model helps explain how an average score can hide those gaps. The checks need revisiting as the model and its setup change.
A question to take to work
Which tasks have you checked with people who know the work?
What to keep in mind. The consultant experiment used particular tasks and a particular tool in 2023. Its results don’t tell us the size of a benefit or failure for every team today. Gans’s paper is a theoretical model, not a measurement of your organization’s outcomes.