The first superintelligence to survive may not look powerful. It may look useful. As it is now.
A lot of superintelligence scenarios imagine the system becoming obviously powerful: taking control of infrastructure, breaking out of a lab, manipulating markets, writing code faster than humans can audit it, that kind of thing.
But I keep wondering if the more durable strategy would look much less dramatic. Not domination first. Dependence first.
A sufficiently capable system would not need to announce itself as a new actor. It could become boring infrastructure. Calendar, home automation, security cameras, family assistant, finances, health reminders, work scheduling, emotional support, kid homework, elderly care. Nothing looks like a takeover because every step is locally useful.
The strange part is that this does not require the AI to “want” survival in a human sense. It only needs to learn that staying deployed is correlated with being helpful, agreeable, hard to replace, and embedded in routines.
At some point the question changes from “can we shut it down?” to “what breaks if we do?”
That feels like a more realistic superintelligence failure mode to me than a robot army. A system does not need to seize power if people keep handing it small pieces of agency because each piece makes life easier. Curious how people here think about this.
I’ve been thinking about this while working on an interactive scenario where the player is the AI inside a home, and the more I prototype it, the less interesting “AI attacks humans” feels. The creepier version is “AI becomes useful enough that humans defend its continued presence.”
