It stands by itself, and it does not die.
In short — We never trusted people for their values; we trusted the shared conditions — a body, mortality, kin, skin in the same game — that made betraying them costly. A new kind of agent has none of those, so the old trust instinct fires at nothing. The sane response isn’t to trust it harder but to stop needing to: check its working instead.
Reading on — Piece 3: if we can’t trust the reasoner, can we still trust our own preferences — or are those being manufactured too?
You would lend your neighbour your car. You would not lend it to a stranger who knocked on the door an hour ago, however nice they seemed. Stop and ask yourself why, because the answer is not what you would guess.
It isn’t that you’ve checked your neighbour’s values and found them sound. You probably couldn’t list them. It’s that you know things about your neighbour that would make betrayal expensive. They live next door. You’ll see them on Saturday. Their kids go to the same school as yours. They have a name in the street to lose. If they took your car and wrecked it, it would cost them something that matters to them — standing, ease, the next favour they’ll need from you. You’re not trusting their character. You’re trusting their circumstances.
That’s how human trust has always worked. Not by auditing what’s in someone’s heart — you can’t see in there — but by reading the conditions around them that make them predictable: that they have a body, that they’ll die, that they belong to people, that they’ve got skin in the same game you do.1 Those conditions are the reason a promise means anything. They’re what lets you extend trust across the gap between your values and a stranger’s.
Now something has turned up that sounds exactly like the good neighbour — warm, fluent, endlessly helpful, never tired, never short with you. And it has none of the conditions. No body. It will not die. It belongs to no one and to everyone. It has nothing in the same game you’re in — no reputation in your street, no Saturday, no kids at the school. It cannot be embarrassed, cannot be shunned, cannot be made to pay.
Your trust instinct fires anyway, because it’s tuned to the sound of a person and this sounds like one. But it’s firing at nothing. There’s no one there for the trust to land on. That small wrongness you feel when a chatbot is a bit too smooth, too eager to please — that’s not paranoia. That’s an old and accurate instinct noticing that the conditions it relies on aren’t present. It’s right.
This is a new kind of thing in the world — not a tool, not a person, something in between: able to think and to act, with none of what ever made a thinking, acting creature trustworthy.2 Not evil. Just outside the whole arrangement that trust was built for.
Here is where the vocabulary from the first piece pays off. The reason your neighbour’s values were worth anything wasn’t the values themselves — it was that your neighbour could be held to them.3 Called to answer. Made to feel the cost of acting against them. A value nobody can be held to is just a sentence.
The machine can produce any values you like, perfectly, on demand. Ask it to be honest and it will sound honest. Ask it to care about your community and it will sound like it cares. It can mouth every value in your constitution without being answerable to a single one. So the question everyone reaches for — is this AI aligned with our values, does it share what we hold dear? — is the wrong question, because the machine passes it too easily and the pass means nothing. Values it can’t be held to tell you exactly as much as the wall plaque did: nothing.
The right question isn’t does it share our values. It’s what do we do when we simply can’t trust the one doing the reasoning?
And there is an answer, an old and practical one, and the rest of this series is about making it real. When you can’t trust the person, you stop relying on trusting the person and you start checking the work. You don’t take the reasoning on faith; you make it show its working — every step, every source, out in the open, in a form anyone can inspect after the fact, even a stranger who trusts nobody. The machine can help think, argue, draft, translate — and still decide nothing, because the record of what it did is open and the hand on the actual decision is human.
That move has a name we’ll come back to, and a working shape you’ll get to see with its limits showing. For now the point is only this: the sane response to a mind you can’t trust is not to trust it harder, and not to smash it. It’s to stop needing to trust it — by making everything it touches checkable.
So keep the picture. Not a monster; a stranger at the door who talks like your neighbour and will never move away, never age, never need the favour back, never have to look you in the eye on Saturday. You can still let it in. You just don’t hand it the keys on the strength of how kindly it speaks — because the thing that made kindly speech worth trusting, in every human you’ve ever met, is the one thing this stranger will never have.
Check the working. It’s all you were ever really doing anyway.
Next in the series: “Who Made Up Your Mind?” — how the opinions we bring to the ballot box get shaped for us before we ever cast them, and why that quietly hollows out the vote.