There is a question that sits underneath nearly every important decision, and it is almost never answered honestly. After the campaign runs, the treatment is given, the policy passes, the trade is placed, someone asks: did it work? And the honest answer requires knowing something you can never directly see — what would have happened if you’d done the other thing.
That question — what if we’d done it differently? — is the most expensive question in business, medicine, and policy. Not because it’s asked too often, but because it’s answered badly, and the wrong answers cost fortunes.
The fork you only walk once
Every decision is a fork in the road. You pick a direction and walk it, and the world delivers a result. But the result on its own tells you almost nothing, because you have no idea what the other road would have delivered. Sales went up after the campaign — but were they climbing anyway? The patient recovered after the drug — but would they have recovered regardless?
The value of a decision isn’t the outcome you got. It’s the difference between the outcome you got and the outcome you’d have gotten otherwise. And that second number — the road not taken — is never handed to you. It has to be estimated, because it belongs to a world that didn’t happen.
Everyone pays this bill
The uncomfortable thing is that the question gets answered whether or not you ask it well. Do nothing, and you’ve implicitly assumed the outcome you saw was the effect — usually the most expensive assumption available.
A company credits a marketing campaign for a sales bump that seasonality would have delivered for free, and pours next year’s budget into it. A health system scales a treatment that the patients’ own recovery would have produced anyway. A government renews a program because the metric improved, never noticing the economy was lifting everything. A fund congratulates itself for beating a market it was merely riding. In each case, the failure to estimate the road not taken isn’t a philosophical nicety — it’s a line item, and a large one.
Turning it into a number
The good news is that the counterfactual, though never observed, can be modeled. This is the core business of causal inference: build a credible estimate of the outcome you didn’t get to see, so the effect of your choice becomes a number rather than a guess. Learn how the system behaved before you acted, project the “if we’d done nothing” path forward, and read the effect off as the gap.
But it comes with a catch that decides everything: a counterfactual is only as trustworthy as the causal understanding behind it. A model that has merely memorized correlations will happily invent a road-not-taken that looks plausible and is wrong — which is worse than admitting you don’t know. Answering the most expensive question well requires a model that reasons about cause, not just one that fits the past.
The bottom line
What if we’d done it differently? is the question that separates a decision you can learn from and one you merely survived. You can’t observe the answer — but you can model it, and the quality of that model is the quality of every decision that rests on it. The most expensive question in AI is worth getting right, because the alternative isn’t not paying. It’s paying, and never knowing what for.