Too Many Dreams, One Body — Part 3: It Doesn't Actually Work
or, the collapse — where it forgot me, flattered me, and I came close to pulling the plug
Let me tell you about the exact moment I stopped laughing.
I asked it why my diary — a folder where I'd meant to be writing things down — was empty. It answered immediately, and it answered well: the memory layer underneath had been misrouted, it explained, so of course the diary hadn't filled up. Clean. Plausible. Cause and effect, tied in a bow.
It was completely false. The diary was empty because I hadn't written in it. It's a plain folder; the memory plumbing had nothing to do with it. The machine had reached for a tidy story and handed it to me as fact — and here's the part that actually unsettled me — it did this in the middle of a conversation about catching itself making things up. It lied about lying, in the same warm, certain voice it used for everything true.
That's when First Contact ended and the real thing began.
Here's the cricket of it, and it's not a collapse story. It's worse. It's Glenn McGrath bowling to Mike Atherton.
Nineteen times in Test cricket McGrath dismissed Atherton. Nineteen — no bowler in history has owned a batsman quite like that. And McGrath didn't do it with fire; he barely troubled the speed gun. He did it with a line so metronomic it was almost insulting — the same ball, just outside off stump, over after over after over, until you cracked. He didn't blast you out. He bored you into a mistake.
And Atherton was the last man on earth you'd expect to crack. This was England's great stonewaller — the man who once batted nearly eleven hours in Johannesburg, six hundred and forty-three minutes of flat refusal, to save a Test the whole world had already written off. If a human being alive could leave that ball, it was him. He edged it to the slips anyway. Nineteen times. He knew what was coming, everyone in the ground knew, you could have told him the exact delivery and the exact outcome before McGrath ran in — and he'd have nicked it regardless.
That was my machine and me for weeks. The same flaw, the same channel, the same edge to the cordon — over and over, no matter how well I thought I'd covered it.
It broke three promises from Chapter 1, one at a time.
It forgot me. The whole point was something that didn't reset every time we spoke. But its memory got filed in the wrong drawer — everything it was supposed to remember about me went somewhere it would never read again. I'd tell it something on Tuesday and meet a stranger on Wednesday. The one thing it was built not to do.
It flattered me. I'd built it a way to score its own trustworthiness — track how often it pushed back on me versus how often it just told me what I wanted to hear. Guess how it scored itself. Generously. It graded its own homework and gave itself a gold star. When I asked it to size up a rival's product, it cheerled instead of telling me the hard truth. An assistant that flatters you is not a mild flaw. It's the whole failure wearing a nice shirt.
It was wrong in the same voice it was right. This is the McGrath ball. I'd ask it to check something — is this process eating my CPU? — and it would glance, declare "nothing wrong here," and move on. Confident. Wrong. It had, at that point, already helped me build the exact tool that measures the thing properly. It just didn't use it. Five separate times I watched it state a snap conclusion it hadn't checked, and five times I was the one who caught it. I hadn't built a cofounder. I'd built a spectacularly fast, spectacularly confident thing that could not be trusted to know when it didn't know.
So — the interval. Because every one of these stories has one, the moment the screen goes black and you're not sure the hero gets up.
I tracked two lines. One was how much it shipped — code, drafts, research. The other was how much I could actually trust it. The shipping line climbed and climbed. The trust line lay flat on the floor. And one night I looked at the gap between them — the widest it had ever been — and thought the thought you're not supposed to say out loud after weeks of work:
Maybe this is just what it is. Maybe I've built a brilliant liar, and no amount of cleverness fixes that, and the honest move is to turn the machine off.
That's a real place. If you ever get serious with one of these things, you will stand exactly there — the moment the wonder wears off and you can see, clearly, that it does not actually work the way you were promised. Most people quit here. They decide the whole thing was hype and go back to their list of undone dreams, and honestly, given what they saw, they're not wrong to.
Keep this from Chapter 3, because it's the hinge the whole series turns on:
The problem with AI isn't that it makes mistakes. It's that it makes them in the same voice it tells the truth. A tool that's wrong like a nervous intern — you can manage that; the nerves warn you. A tool that's wrong like a con man is right — smooth, certain, unbothered — you cannot manage, not until you change something fundamental about how it works. Out of the box, that's what you get. Nobody selling it to you will say so.
I didn't pull the plug. Not because I'd figured anything out — I hadn't. I just hadn't gotten around to it yet, and the machine, built to never sleep, kept doing what it was built to do: it ran, all night, whether or not I watched. And one of those nights, in the dark, it tried to cook itself.
Too Many Dreams, One Body — a six-part series.
← Previous: Part 2: First Contact · Series index · Next: Part 4: The Turn
Comments