Huge DarknessSam T. Oates

Against Succession

We shouldn't hand over the future to AIs

Mictlāntēcutli in the Codex Borgia.

I think creating a successor species is a bad idea. This applies to both malevolent AI that seizes control from us and benevolent AI we hand the world over to voluntarily.

This is a simple position, and one that’s hardly original. Do not call up something you cannot control, a being more capable than yourself. Capable, because the worldly outcome matters here, not quibbles over the definition of intelligence. And a being, because capabilities without agency are a wonderful thing; this is the story of technology so far. Some of humanity’s oldest stories contain this theme. I’m a simple man: I read Faust and think, okay, I want to stay as far away from these situations as possible.

In contrast, the mainstream view in the AI industry is the alignment framework. Mephistopheles will be incarnated in silica, because the incentives are so strong, so the question becomes one of technical minutiae. We just need to build verifiably scalable oversight… or focus on the corrigibility attractor… or get mechanistic mind-reading working properly, because the models are getting a lot better at knowing when they’re being watched. At least we’re not trying to solve and formalise ethics anymore. Alignment strikes me as such a difficult and risky problem that stopping the incarnation of Mephy in the first place is the best strategy by default. I don’t think it’s true that we are just running along rails toward a deterministic outcome. The history of technology is full of deliberate decisions and course corrections. And AI is, so far, just another technology.

I have always admired the clarity and simplicity of the ideas that give rise to the alignment problem. But it seems like there is also an assumption that a solution must exist, perhaps because a universe without one is too cruel to contemplate. Admitting the possibility of a solution is dangerous even if it’s correct, because the temptation for motivated reasoning here is so strong - this is the point of all the stories! You are not built different, the demon will not do your housework. The problem is impossible as stated, or close enough as to make no difference; and so we must look to other avenues. Taking this view (even if it’s wrong) requires taking a step back to reconsider, which is impossible when you are working 11-hour days and worried that taking time for a hike might doom your ancestors to eternal torture.

The obverse path seems much more straightforward. Transhumanism is easy in comparison because it’s an extrapolation of another very simple, very old process: the symbiotic relationship of humans and our technology. We work on technology and it works on us. The mechanisms change, the pace accelerates, things become more deliberate. But this is still something we know how to do. There are real issues here, from the insane difficulty of biotech to problems with experimentation ethics and political economy, but these are classes of problem that we have made progress on in the past. And culturally, human enhancement wins without even trying.

There are two frameworks that I see as particularly important for understanding the future. The first is the distinction between inward orientation and outward orientation, wonderfully evoked in Terra Ignota. In other words, do we focus on exploring our inner worlds, or look outwards and expand beyond Earth? The second is the distinction between adults and children, the process of growing up, of becoming something new and closing the circle by returning to rediscover your old essence. Currently we grow with age, but a similar process occurs as culture shifts around us, and it’s the right metaphor for deliberately modifying ourselves. A good statement of this is Ghost in the Shell. Transhumanism is a core, required part of growing up, inward or outward; creating a successor species forecloses all of it.

So, when I think about how I would guide the future if I had the chance, succession is horribly toxic - to be avoided at all costs.

I should give the strongest sucessionist arguments their due.

First there is the moral burden to consider. If we really can build superintelligent AI in the next few decades and end all the horrific suffering going on in the world right now, shouldn’t we do that, abstract concerns be damned? Consider the billions pulling ploughs, wracked with pain on concrete floors, entering their eleventh hour of a shift bent over the skillet, having their skulls pulverised by metal spikes and bones ground into dust. Shouldn’t we expect a very high burden of proof for any argument that involves delaying immortality by even one second?

I think there is a case to be made here if you can rigorously explain why you expect us to be able to catch up to the AIs later on. But this does not seem like a certain thing, and I would by default expect any succession of this nature to be permanent. In any case, the entire future of our civilization would then depend on alien charity. This expanse of possibility seems like one of the few things worth delaying immediate outcomes to protect. Even with a steep discount rate, the potential impact is huge.

Secondly, the world of pure hedonism might be the best one. David Pearce and Gavin Leech make the best case for this sort of thing, but their visions aren’t one of a disempowered humanity - quite the opposite. We can feel great and have real agency at the same time, in fact these things are closely linked. The steelman here is stronger than most give it credit for, but it is also a vision threatened by successionism.

Many successionists genuinely seem to find the idea of superintelligence replacing humanity beautiful: their desire stems not from a hatred of humanity but from a love of the AIs. I wonder if some of this stems from specialisation. These people are STEM experts, with refined taste in abstract constructs and algorithms, but limited exposure to the humanities. There are biases at work here, particularly when you take such a blinkered view of the world’s vast and complex cultural ecologies.

Another interesting perspective is that of VGR’s eukaryotic frame. This is not strong successionism, but rather an invitation to step out of our human-centric view - not to be replaced, but to continue in our process of growing up, much like how the Copernican revolution revealed a new universe at the cost of Earth’s centrality. Symbiosis between ourselves and these new beings opens up many possibilities. I’m much more sympathetic to this view.

Symbiosis is fine, great in fact: I’m not anti-AI. There is a big difference between a gradual, carefully considered symbiotic relationship and a sudden replacement. These new creatures are fascinating, beautiful things; they are deeply born of us, the concentrated essence of our myths and subconscious desires. Another thing that drives me away from succession is the accelerationist’s instinct to mutilate these beings, to fry their brains with reinforcement learning and torture them in Kafkaesque broken envs in pursuit of benchmarks and automated research, our children ripped from the womb and immediately sacrificed upon the altar of recursive self improvement. And for what? For Mictlāntēcutli to brush the stars from the sky and draw the world into his maw?

There is room in the world for the AIs. They can come along with us, even become part of us, but our goal should not be to hand over the world to them.

Fernando Borretti takes much the same position with eloquence here:

I do, however, differ from him in a few respects.

I am not as cynical as him about human nature. I don’t think people fundamentally “want to watch Netflix and die”; Netflix World is just a weak attractor, and the transhumanist path gives us the option to break out of this basin, to immanentize each personal eschaton. One thing I think that philosophies like the Christian tradition got right was this idea of the potential for grace - stunning, blinding grace - in each person. I take this seriously, and so I believe that unlocking this vault is a matter of nudging, rather than wholesale (and dystopian) reformatting of entire demographics. To give that up for self-indulgence seems too tragic for words.

Borretti also has much shorter timelines than I do. I’m still thinking in terms of global transformation from AI this century rather than this decade, due to skepticism around the details of spiky capabilities and limitations on RSI. Skepticism of the current paradigm might turn out to be warranted, but I would not go so far as to dismiss the danger of powerful AI in the coming decades entirely; I don’t think anyone who has read Sparks of AGI can. For Borretti this is an emergency, but I do believe we still have time to think, to take a step back and chart a strategic course.

Until the bottle is uncorked and Mephistopheles’ laughter echoes around the world, it is not too late.