This blog has permanently moved to substack! You can find it at https://nothingismere.substack.com.

A sample blog post from the new Nothing is Mere, now with less mereness than ever before:

A Near-Term Policy for Not Getting Killed by AI

Hundreds of scientists, including three of the four most cited living AI scientists, have said that AI poses a very real chance of killing us all.

We’re in uncharted waters, which makes the risk level hard to assess; but a pretty normal estimate is Jan Leike’s “10–90%” of extinction-level outcomes. Leike heads Anthropic’s alignment research team, and previously headed OpenAI’s.

This actually seems pretty straightforward. There’s literally no reason for us to sleepwalk into disaster here. No normal engineering discipline, building a bridge or designing a house, would accept a 25% chance of killing a person; yet somehow AI’s engineering culture has gotten to the point that no one bats an eye when Anthropic’s CEO talks about a 25% chance of “doom” for the entire world.

A minority of leading labs are dismissive of the risk (mainly Meta), but even the fact that “will we kill everyone if we keep moving forward?” is hotly debated among researchers seems very obviously like more than enough grounds for governments to internationally halt the race to build superintelligent AI. Like, this would be beyond straightforward in any field other than AI.

Obvious question: How would that even work? Like, I get the argument in principle: “smarter-than-human AI is more dangerous than nukes, so we need to treat it similarly.” But with nukes, we have a detailed understanding of what’s required to build them, and it involves huge easily-detected infrastructure projects and rare materials.

Response: The latter points are also true for AI, as it’s built today. The most powerful AIs today rely on extremely specialized and costly hardware, cost hundreds of millions of dollars to build, and rely on massive data centers that are relatively easy to detect using satellite and drone imagery, including infrared imaging.

Q: But wouldn’t people just respond by building data centers in secret locations, like deep underground?

(Read more)