It can map every inch of the mountain. It cannot tell you whether to climb it.
In short — No amount of knowledge tells you what you ought to do: a machine that knew everything still couldn’t say what matters. Its brilliance isolates the values question rather than answering it, leaving legitimacy — the right to decide — more ours, not less. The real danger is handing that over out of convenience; the guard against it is a plain rule — the software rehearses, people decide.
Reading on — Piece 5: so what does a room that keeps people in charge, and hosts real disagreement, actually look like?
Think of the cleverest person you know. Not the loudest — the genuinely wise one, the one whose judgement you’d trust with something that mattered. Now take them a real decision. Should you leave the job. Whether to have the child. What to do about your father. Watch what the good ones do: they ask questions, they lay out what’s at stake, and then, at the crucial moment, they hand it back. That’s yours to decide.
They aren’t being coy. They’ve hit a wall that no amount of cleverness gets you over — and it’s the same wall that stands between the most powerful machine ever built and the one thing we’re all quietly afraid it will take.
A philosopher noticed it three centuries ago and it has never been knocked down: no pile of facts, however high, tells you what you ought to do.1 Knowledge describes what is — how the world works, what will happen if you pull this lever. It is silent on what you should want. You can know everything there is to know about a river and it will not tell you whether to dam it; that turns on what you care about, and caring is not a fact you can look up.
This sounds abstract until you put a superintelligence on the other side of it. Suppose a machine arrives that knows more than all of us combined and reasons better than any of us ever could. It hits the same wall. Ask it what we should do — with our economy, our coastline, our old age — and it still faces the question no cleverness answers: in service of what? It can tell you, faultlessly, how to get anything you want. It cannot tell you what to want. That was never a knowledge problem, so being the best-informed mind in the world gets you nowhere with it.
Here is the turn, and it’s the steadying one. We assume that a mind far above ours would take everything — including the question of what matters. It’s the opposite. The machine’s brilliance strips away every part of a decision that was a knowledge problem — the forecasting, the modelling, the sorting of options — and leaves standing, alone and floodlit, the one part that never was: the authority to say what’s worth wanting. Superiority doesn’t colonise the values question. It isolates it. It clears everything else away and hands you the one thing that was always yours.
There are two different things here, and the whole century of argument turns on not confusing them.2 One is capacity — the ability to build, run, calculate, decide well. The other is legitimacy — the rightful standing to be the one who decides. A machine can run away with capacity and not gain an ounce of legitimacy, because legitimacy isn’t a skill you can be out-performed at. It’s a standing, and it sits with people, with tikanga, with the room that has to live with the result. As the machine takes more of the capacity, the legitimacy becomes more ours, not less. The most important thing is the safest.
So the threat was never a machine seizing power by right. It’s subtler and closer to home: that we hand the power over ourselves, gratefully, because its answer is better argued than ours.3 Faced with a hard call and a system that produces a confident, well-reasoned recommendation, the path of least resistance is to take it — and to forget that “well-reasoned towards what?” was the part only we could set. No one storms the citadel. We leave the gate open and go home, because deciding is hard and the machine is so very good.
That’s the surrender to design against. Not rebellion — abdication. And it’s designed against not with a firewall but with a habit, built into the tools themselves.
The habit has a shape, and it’s the spine of everything this series is heading towards: let the machine do what it’s genuinely good at, and stop it doing the one thing it must never do. Let it rehearse the argument — lay out the options, make the strongest case for each side, find the fault-lines, flag the weak evidence, provoke. And then let it stop, and hand the decision back to people, the way the wise friend does.4 The software rehearses; people decide. That single line, built into how the thing works rather than printed on the box, is what keeps a reasoning machine as an instrument of human judgement instead of a replacement for it.
A machine can be the best-informed voice in any room. That is a gift, and we should use it. It just isn’t a vote, because it can answer every question except the only one that finally matters:
In service of what?
Next in the series: “A Room Built for Disagreement” — why lasting disagreement between good people over good things is not a failure to be fixed, and what a decision-making room looks like when you stop pretending it is.