Casey Newton, whose fiancé works at Anthropic, writes in response to Mark Zuckerberg:
On Sunday night, HBO aired the third-season finale of House of the Dragon, the stuffy and unrelentingly bleak prequel to Game of Thrones. The show takes place in a world where one great house holds a near-monopoly on a superweapon — dragons — and thus rules the world. … In Westeros, no one is confused about what a dragon is; in all likelihood one has been responsible for the death of someone they know. But here in our world, where the technology industry is investing trillions of dollars into building and serving ever-more powerful artificial intelligence models, we struggle to imagine the end state.
…
In the long run, though, the most worrisome element of Zuckerberg’s manifesto is the way it tries to redefine AI safety as a question of power distribution rather than control over the systems that we are building. Whether we can control AI systems is no longer a theoretical question, given recent revelations about OpenAI models scheming against the company for months on secret message boards (see below). And yet to Zuckerberg, the danger isn’t that we might lose control over these systems — it’s that someone else might monopolize them.
It’s noteworthy that Casey Newton has to use Game of Thrones, a fantasy series, to come up with an analogy to support his point.
Here’s an analogy based on reality:
Casey Newton’s fiancé works at a company which stands to make its employees millions or billions if it succeeds in monopolizing AI. Casey Newton argues that we must allow this to happen because otherwise AI will create Jurassic Park, or some other kind of doom. This is like a member of the feudal nobility arguing that peasants shouldn’t be allowed scythes that are too sharp. Otherwise they might hurt other peasants! Much safer for the lords and the knights to control sharp things and for the peasants to stick to farming.
If we’d allowed that kind of logic to prevail in the past, 90% of the population would still be in the fields, farming so that Anthropic employees—I mean, feudal nobility—could write doggerel about how they saved the peasants from scythe doom.