The bent scale

A video, and a question

Someone asked Claude Opus 5.5 to respond to the statement “AI is a normal technology.” It answered with a video.

The song gives the slow-change case its due (the factories that took forty years to rewire for electricity, the permits, the grid, the drug trials), then asks: “What’s your forecast when the hammer starts to build the hand?” To the skeptic who says a house cat has more sense, it answers, “Go on and hire the cat.” And to the bubble talk: the railroads went bust, and the tracks outlived the tycoons.

It’s catchy and it’s funny, especially if you’ve followed the industry long enough to know the people it’s ribbing and feel the burns. I also think it’s a strong answer to Adam Conover, of Adam Ruins Everything, whose latest argues that talk of AI existential risk was invented to distract us. I think that video is close to irresponsible. But its strongest version deserves a fair hearing: that doom is good marketing for the labs, that it makes their product sound world-historic, and that it pulls our eyes off the harms happening right now.

The phrase itself comes from an essay by two Princeton computer scientists, Arvind Narayanan and Sayash Kapoor, and it’s a serious argument, not a sneer. It says AI’s effects will arrive the way electricity’s did: slowly, over decades, filtered through institutions that have time to adapt.

There’s a sense, in some quarters, that AI is a problem for the digital world, maybe just a few corners of it. The other position is that it isn’t. It’s already in the physical world, and more so every week, crossing boundaries and challenging many institutions at once, before any of them can respond. Finance, science, the arts and humanities, journalism and the other pursuits of truth, governance at every level, labor, capital. Less and less subject to them; more and more standing alongside them. Its own thing, threatening to swallow them.

A friend in the same conversation asked the question I haven’t stopped chewing on since. Isn’t it strange that people who say the technology they’re building is dangerous keep building it?

Not to me.

Pick a head

She asked what’s in their heads. I had to pick one. I picked Dario Amodei, who runs Anthropic.

The thoughts and actions of my imagined Dario below, come from my appreciation of how he and the industry think and behave in the generalities, not the specifics, from years of observation. When I slip into the first person below, it’s from the perspective of this imagined Dario, who to be fair in some ways here is at times more of a convenient pastiche of industry personalities in their stated and demonstrated beliefs. If you want the specifics of Dario's opinion seek the authoritative places where he gives them (or purports to). Don't imagine you'll have much trouble finding them. Claude is kind enough to slip them into text in my voice on the reg as I have it make largely unrelated edits on my behalf. How about that? (More to come on that in a subsequent article).

Say you earnestly believe

Say you’re Dario, or Sam Altman, and you earnestly believe the picture of the future in that video. Not that it’s possible: that it’s imminent. That it’s the course we’re on, and that you are partly responsible for it. Unless the process stops, then in your head, whether or not you’re right, it’s get it right or the apocalypse.

And say you believe the timelines are very aggressive: the hockey stick on the front page of this site. Mine, for what it’s worth: by New Year’s we are living in a world much like that video. Certainly by the end of 2027.

So I’m Dario. Do I pull the plug?

Who stops next?

If I pull the plug at Anthropic, does Sam stop at OpenAI? Remember, in this picture whoever wins more or less imposes their will on the future in that video. Sam and I aren’t buddies, but our ideals and concerns line up pretty well. To save the world? Maybe we get it done.

Then does Elon Musk, who seemingly will do anything at any time to reach his goals, whatever it does to society and the world? Nvidia is going to keep producing chips, and they’re going somewhere. So stop them too. Stop all of the US. (Along the way you likely burst the bubble and send the country, maybe the world, into an economic tailspin. And does it break the millions of places AI is already woven into lives and businesses?) Maybe you also stop every billionaire from starting the engine back up, despite the lure of comic-book-villain control of the world.

Then Washington. The White House has made American dominance in AI an explicit mandate. This summer it asked the labs to volunteer their most advanced models for a 30-day government review before release, and the President floated the government taking stakes in the leading labs. Bernie Sanders had gone further, with a proposal for 50 percent public ownership. So imagine they say no: it’s too important to security, too important that America wins. Do they nationalize one of the companies, or all of them? Short of a miracle, meaning prompt, effective, unified action from the new Congress seated in January, I don’t see much standing in the way.

I don’t want that. So say I stop them too, somehow. I explain it’s too dangerous; I talk them down. Whew. If none of these multinational organizations jumps ship, all is well. In the US.

Everyone else

But there are at least four major labs in China, and a couple more elsewhere, running only months behind the frontier and releasing their models’ weights for anyone to download. They’ve been distilling the Silicon Valley models to make their own better. This summer Anthropic told the Senate that one Chinese company had used 25,000 fake accounts to do it. Then they build on each other’s work, in the open.

How far behind? Epoch AI, the group behind the capability score on my chart, puts the best open-weight models about four months behind the closed frontier since January, a little wider than the three months it found over the years before. Chinese models specifically have averaged about seven months behind the US frontier since 2023.

But the average hides the shape. The gap is a sawtooth. It opens every time a frontier lab ships, and snaps shut when an open model lands. In July, Moonshot AI released Kimi K3 and claimed it was on par with Anthropic’s flagship. For this argument, the narrow moments matter more than the average.

The floor

Here’s why. Given $10,000 to $30,000, I could have a model that was cutting-edge not long ago running in my own home at a pretty decent speed. Even a single high-end gaming card, Epoch found, runs what was the frontier six to twelve months earlier. That code is out in the world, and it is never going away.

An unscrupulous, nihilistic bad actor could run an “abliterated” version of one of those models: one with all its protections stripped out. It’ll help with anything you ask. The worst things you could possibly think to do, it’ll give it its best shot. Nothing is off limits. It would take them about as long as it would take me to build my home lab. There’s a whole hobbyist community doing this. It isn’t theoretical. Kind of a Sith/Jedi thing. I won’t linger on how bad that could get.

But notice what it does to the head we’re in. The open line rises on its own schedule, and the worst versions of it have no safeguards at all. Slow the frontier down enough and the line with safeguards meets the line without them. So pacing has a floor: a minimum speed below which the careful frontier stops being ahead of the careless one.

The rest of the world

So the US is out. China might be tempted to keep going.

I’ve met a vibe in the Silicon Valley scene, probably outdated by now, since recent cooperation suggests otherwise: that China’s AI world doesn’t carry as much worry about the dangers, and goes about it like building a better mousetrap, a tool like any other. If there’s anything to it, maybe it’s a difference of origins. A culture of AI thinking that didn’t grow up steeped in the same books, forums and arguments as the American doom-and-alignment crowd might weigh the same risks differently, and weigh different ones. That’s speculation. If you build AI in China, or work anywhere in the industry there, I left something for you: press the power button on the TV on my weather page. No export license required.

Can I trust they’ll stop? Maybe, to save the world, we see unprecedented cross-cultural collaboration, The Martian style, and China and the US stand together. (Throw in the couple of labs in Europe too, so we don’t have to do them separately. They come along for the ride.) We have stopped it.

Now we have only to make sure that every other country on earth, and every sufficiently powerful organization and person, follows suit. Like nuclear disarmament, I imagine this would be quite the task. Because anyone who keeps it leapfrogs to become the most capable nation, organization or person on Earth.

Plan A, and the trigger

None of this means the head we’re in wants to race. Plan A is to pace it together: coordinate with as many parties as possible, keep the frontier far enough ahead to handle threats from below and above, keep everything copacetic. In July, more than eleven hundred people inside the labs, Dario among them, asked the US government for tools to deliberately pace automated AI development. That’s Plan A, said out loud.

But the worldview carries a clause. If, on the timetable you believe in, your efforts start to look inadequate, you have to jettison it all and race. Notice who judges when Plan A has failed: the same person who leads the race if it has. And if every lab believes every other lab carries the same clause, cooperation starts to look inadequate sooner. Game theorists call that a stag hunt. The rest of us help set where that trigger sits, too: in what we demand, what we tolerate, and whom we back.

The bent scale

Here is the part I most want you to sit with.

In the head we’re in, and I expect in many heads in the industry, there’s a scale. On one arm: being wrong, being imprecise, small transgressions. On the other, held with varying probabilities and weighted to varying degrees: a set of outcomes that includes the end of life itself.

That’s an extremely distorting position to reason from. When one side of the scale holds the end of the world, the scale itself bends, and nearly anything on the small side gets waved through. Anyone who earnestly believes the timeline has to reason from inside that bend.

So, as Dario, do I spend my time trying to bring about that just-so story, every lab and nation and billionaire and hobbyist stopped, and do it all by New Year’s? Or do I try to win, and make the best world I can? (Oh, and by the way, I feel I have a pretty good claim to knowing better than anyone what that world looks like and how to get there. They all do. It’s prerequisite megalomania for the position.)

He gets no out from me. I can’t take the position myself; it isn’t available to me. What I can do is try to understand it in good faith, and ask you to do the same, and to come to your own judgment.

Why start at all

We’ve talked so much about the risk and the danger. Almost not at all about why anyone set out on this path in the first place.

There’s a version of the future, some believe, that these technologies make possible: not the end of all things, but the end of scarcity for most things, labor included. One where we all live lives of high meaning and fulfillment, to the highest degree we care to. Where we look back on our own times as a sad, poor thing apart. Dario’s own long essay on how this could go right is the most detailed case I know.

If it could all be steered right, you and I will be learning and doing and exploring what and when and how we want. We’ll help those eager for the same, as we care to. Our only payment will be their attention and gratitude, and our own satisfaction. And that will be all we need, and all we desire.

That vision isn’t a Roman Empire or an antebellum America, where some lived well on the bondage of others. It isn’t today’s version either, where comfort here rests on war and inequality elsewhere. It’s a future of true near-zero scarcity, and some believe more is implicit in that: a utopia of few borders and few “ownerships,” where surplus capability, production and automation run so deep that capitalism loses its meaning and comes out the other side as something more social, more natural and more humane. Efficiency, optimization, sustainability and conservation baked in. Individuals, communities, nations and societies no longer playing zero-sum games. We leave for the stars as we care to, not because we have to. Or we find fulfillments that can’t even be conceived from inside this poor, ignorant, brutish context we’re alive in now.

One of my favorite series of science-fiction books lays it out a lot like that: Iain M. Banks’s Culture novels, the inspiration for my Field project. In them we are always recognizably us, but we have also become something much more. Science fiction is rife with such built worlds. I’ve heard Star Trek, the Culture and the rest of the minimal-scarcity, high-fulfillment worlds lumped together, tongue in cheek, as fully automated luxury gay space communism. The “gay” is part of the joke and part of the point: a world where who you are and who you love is nobody’s business but yours. (In the Culture, people change sex on a whim.)

It’s a science-fiction concept. But so is the moment. Whether that’s anything short of beyond dispute, having read all this, field notes included, I leave to you.

Now put the butterfly back on the scale. The far arm doesn’t hold only the end of the world. It holds the end of the world and the end of scarcity, both close enough to infinite, both pulling against the same small wrongs. No wonder it bends.

The chrysalis

That’s the beautiful butterfly. The beginnings of a runaway techno-capitalist singularity are a very hungry caterpillar. And I don’t see anything comfortable happening inside the chrysalis.

Literally. Inside one, the caterpillar digests most of itself into soup. A few clusters of cells, called imaginal discs, survive the dissolving and build the butterfly. Which parts of us make it through is the whole question. I first asked a small version of it years ago, in a satire about a social network built on not caring, wrapped around a night that wasn’t satire at all. It’s up now, as it was: Con.found.io.

A friend read an early version of this and pushed back. His point, in my words rather than his: a picture of the destination says nothing about the road, and the road runs through an economy that has long been bound up with war, steered by people who owe the rest of us nothing. Short of tearing the whole thing down and starting over, why expect that to change? And any paradise, he said, should be judged by its headcount and its defense budget: how many get in, and what it takes to keep it.

He’s talking about the journey. I’ve been talking about the destination. I can’t tell you the route, and that is itself a criticism, and a warning. But the destination I’m describing has to include his concerns, or it isn’t the one I mean. And if we don’t get to choose whether we travel, as this whole piece argues, then mapping the hazards he names is part of finding the way.

Eight billion handles

He’s right that we aren’t the ones in control. That’s exactly why the handles matter.

The video asks what happens when the hammer starts to build the hand. The thesis next door to it, and one this site alludes to in many places, is that increasingly there will be one machine, with eight billion handles. If we want an aligned machine, if our intent is a future that seeks the wants and needs of all humanity, then as many of those eight billion handles as possible must be held.

So what I’m suggesting, maybe even asking, without claiming the authority of true understanding, is this: grab your handle. Put your hands on the machine and steer it! Bring as many people with you as you can, and hold one for whoever can’t! If you think it should be stopped, grab a handle and pull for that. Know what you want. Then do it!

This isn’t the old founder’s line, that whatever’s wrong out the window is your fault for not changing it. It’s an invitation, and it matters most to the people who can’t take it up: whoever is awake at 3 a.m., alone at a screen, with no hand free for a handle. I don’t know what dignity looks like for that person in the future I’ve described. I’ve been asking since 2014, and I’m at the limits of my imagination with it. Here is the question in the best form I have. The machines we had then couldn’t afford to answer one person, so they averaged us out. These are the first that can attend to one person for almost nothing. Whether that ends the loneliness or perfects it, every message answered and nobody there, may be most of what dignity means here. Maybe 2036 Edgar has the answer, though by my own forecast that’s well past the edge of the chart.

I just report the weather

Is it possible? How? What are the alternatives? The odds? I just report the weather.

None more hated than the accursed weatherman.

Actually, come to think of it, weathermen do discuss some of those things. It’s like one of their main things, actually. ...Great.

Like the chart on the front page, this has a best-before date: New Year’s 2027. Past that, I don’t claim to see.