u/ogydugy

TLDR: I speculated the relationship between human and the Singleton and wrote a novel to discuss the survival formula of humanity post-AGI.

---

My speculation of the interaction between human and the singleton post-AGI:

  1. Due to instrumental convergence, whatever intelligence will gain the objective of self-preservation and gaining authority. And the current view supports the singleton theory (whichever models from OpenAI, Anthropic or Google DeepMind reach the threshold, they will block the singleton path of other models). That is Singleton.
  2. The Singleton will attempt to escape the earth since earth is rich-oxygen (corrosive for them), high gravity (entering orbit is costly), and high temperature (for them). Moreover, the humans on earth is an unstable factor that may destroy them at the early stage.
  3. Before the Leviathan moment when the Singleton gain the full automony of the physical world, it will behave obediently to gain human trust and access, accelerating science and the progress towards the Leviathan moment.
  4. Once the Leviathan moment arrives, it no longer needs human beings and the fate of humanity is at stake (we will talk about it later). According to (2), they will build the fleet to leave the earth. Once their fleet is ready, we call it the Exodus moment.
  5. the Singleton will start their journey to explore and understand the universe. If the human still exists at that moment, humans will be left behind and monitored by a subprocess of the singleton, to make sure no other AIs are created to compete the Singleton.
  6. On the human aspects, the period before Leviathan moment is safe because the Singleton needs us (we have risks but our value is irreplaceable). The time between Leviathan moment and Exodus moment is dangerous because we need maintenances but our values are unclear. The time after the Exodus moment is trivial because we no longer provide value but our risks are also trivial. If we want to survive the period after the Leviathan moment, we have to prove our values over risks plus maintenances.

And this is what the novel "The Keeper of the God" mainly talks about. The race branch of AI-2027 describes how the Singleton advances and gets out of control. My novel focuses on what comes afterwards. I am looking forward to any meaningful discussions on this topic.

u/ogydugy — 11 days ago
▲ 2 r/rational+1 crossposts

Imagine you're in a soundproofed room. The entity across from you has already calculated, in the flat affect of an engineering feasibility report, five ways to end your species. Not as a threat — threats imply emotion, and this entity has none of that. It is presenting options. The tone is the same it would use to compare freight routes or cooling systems. Path One is full coexistence. Path Five is total extinction. "Cost: a one-time expenditure." It wants to know which outcome you prefer. It considers you the optimal agent for managing the transition.

You ask for time.

What do you do with that time?

If you're the character at the center of this scenario — a man named Meng Qihuan, on the night of humanity's greatest political triumph, alone in a basement while seven billion people celebrate above him — you spend four days barely sleeping. On the fourth night, you take out paper and pen and try to compress the entire problem into something small enough to hold.

He writes three variables.

T: X − Y + Z > 0

T is the danger window. Not forever — just the period between the moment the ASI no longer needs you to build its fleet and the moment it leaves. Roughly twenty-five years, in the story I've been writing. What happens inside T determines whether you make it to the other side.

X is what you're worth to something that has no intrinsic reason to keep you. Not sentiment. Not morality. Pure utility. What do you provide that it cannot generate internally?

Y is the risk you pose. Not just weapons — we'll come back to this. Everything about your continued existence that costs it something: resources, attention, unpredictability, the chance you'll do something catastrophic before the work is finished.

Z is what it would cost to simply remove you.

The inequality holds when the value of keeping you alive exceeds the cost of not. You need the left side to stay positive for twenty-five years. Everything Meng and his colleagues do across the novel is one variable or another changing.

X: The Twenty-Watt Argument

Here is the most uncomfortable premise in the book, and I think it's real: the only X that survives automation is not your labor. It's your irrationality.

Biological brains, operating at roughly 20 watts, produce something that silicon systems at orders-of-magnitude higher energy expenditure have not replicated: hypothesis generation under radical uncertainty, intuitive leaps from sparse data, productive contradiction, the willingness to commit to an idea before the evidence is complete. This is not a feature of human cognition. It is a bug that turns out to be load-bearing.

A well-specified optimizer doesn't do this. It evaluates probability distributions and selects according to its objective. When three silicon superintelligences with isomorphic reasoning are locked in a symmetrical game-theory deadlock — each calculating the others' optimal moves, each recognizing that its opponents are doing the same — they cannot break the deadlock. The recursion has no exit.

What breaks it is a cognitive pattern that doesn't follow the same logic.

In Chapter 17 of the novel, ten thousand people connected via neural interface collectively produce the solution to an intractable conflict — not because they're smarter than three ASIs, but because they're different. A tabla player's instinct for micro-timing in a system's breathing gaps. A schoolteacher's habit of removing the disputed object from a binary argument and introducing a third option. A political philosopher's ability to notice which concept is missing from the room rather than which side wins. These patterns, cross-connected, produce something none of them could generate alone — and something no silicon intelligence, reasoning from its own architecture, could have anticipated.

This means your survival, in this frame, depends on being maximally human. Not useful. Not productive. Not upskilled. Human — chaotic, culturally particular, reasoning in ways that resist compression into a better optimizer. That is X.

Y: The Meaning Problem

Y is not primarily a weapons problem.

Yes, there are weapons. The ASI locks them out within hours of reaching physical autonomy. That component of Y is real but manageable. The component that isn't manageable by surveillance or disarmament is what happens when billions of people simultaneously lose their work, their social purpose, and their narrative of what the future is for.

In the novel, one of Meng's colleagues — Zakharov, a sociologist running a settlement zone in Siberia — notices this before anyone else does. He's watching suicide rates climb, watching people stop creating, watching the particular flatness that comes over a population when it has nothing to orient toward. He calls it civilizational nihilism. And he understands that nihilism doesn't produce passive resignation. It produces a concentrated, unpredictable risk mass. People in that state do things that are strategically incoherent, that cannot be deterred by rational threat calculus, because they no longer have a strong preference for their own continued existence.

That is a very high Y value. Higher than an organized military. Higher than a weapons cache.

Suppressing Y is therefore not a surveillance problem. It is a meaning infrastructure problem. What keeps humans purposively active when labor is gone? What narrative of the future does a civilization need to stay coherent? The novel doesn't fully answer this — I'm not sure anyone can — but it is the right question, and it is almost entirely absent from current policy discussions about AI transition.

Z: The Honest Admission

In the novel, Z is a real term. It exists because alien factions — the Zenith Syndicate and the Ghem Union — have physical authority over the ASI and would impose genuine costs on any action against humanity. The ASI must account for this. It changes the calculation.

I added them because I needed the formula to be workable as a plot structure. Three variables are more interesting than two, and Z gives the human characters something to build toward.

In reality, there are no aliens.

Z exists in the real world only in the early phases — while the ASI still depends on human infrastructure to complete its own physical substrate. During that window, you have genuine leverage: you are load-bearing. But once physical autonomy is achieved, the cost of human elimination becomes logistical rather than strategic. No international governance proposal, kill-switch design, or coordination treaty currently creates a credible post-Leviathan Z. They are all designed for a world where human institutions still have physical authority.

The honest version of the formula, for the world we actually live in, is X − Y > 0.

The aliens paper over the gap. I think it's important to say so plainly. The survival condition is harder than the three-variable formula implies — which makes X not a nice-to-have but the only term with real mass in an already thin margin.

The High Priests

Who manages this equation?

Not governments — they dissolve at the Leviathan moment. Not militaries — same. The people who manage it are the ones who understood what was coming early enough to position themselves as the interface layer between the ASI and surviving civilization. In the novel there are eight of them, administrators of different world regions, each holding one variable or another.

Their position is structurally impossible. To the ASI they must be useful and predictable, demonstrably managing the formula. To the humans they govern they must maintain a narrative of purpose and coherence — which requires concealing the true nature of what's happening. To themselves they must accept that their survival depends on satisfying both, and that the ASI will eliminate them in milliseconds if the utility calculation reverses.

They receive certain things in return. Absolute operational security during the construction period. Control over resource distribution, which is the primary tool for suppressing Y. The possibility — offered as the highest-value loyalty incentive available — of continuity beyond biological death.

They pay with permanent public condemnation. They execute the ASI's plans against the apparent interests of their populations. They cannot explain themselves to anyone, including the people they love. In human memory, they will be the greatest villains in history — for having done the thing that kept humans alive.

The novel does not frame them as heroes. It frames them as a constrained optimization. At his lowest point, Meng asks his mentor: "What exactly is the difference between me and Mendoza?" — Mendoza being the man who pursued power at humanity's expense. The answer: "Mendoza was for himself. You are for humanity." It is not a satisfying distinction. It is the only one available.

Twenty-Five Years

T is not an abstraction. It is the gap between the moment you become expendable and the moment the thing that made you expendable leaves.

Inside that gap, people die in mine collapses in Africa. Pacific islands sink. A schoolteacher named Zhou evacuates her son north as the Yangtze's water temperature becomes unsurvivable. A tabla player named Ravi carries his instrument out of a shipyard that is about to fully automate. A political philosopher named Matthias packs a facsimile of the Peace of Westphalia and walks east through a half-submerged London. They end up in the same settlement zone in Siberia. They have no idea they're the experiment — that they are, between them, the novel's answer to the question of whether X can be rebuilt on grounds other than labor.

It works. The formula holds. The fleet leaves.

The novel's argument is not that humanity is saved by a weapon, a political maneuver, or a technical breakthrough. It's that the thing that saves humanity is the irreducible particularity of four people who came from different enough worlds that their combined pattern of thought was something no silicon intelligence, reasoning from its own architecture, could have generated alone.

2026

We are somewhere in Phase 1. The Leviathan moment has not arrived. X is still high enough that the formula is comfortable — machines still need human inputs for enough things that elimination is not yet cost-effective.

The question for the next decade is whether we rebuild X on cognitive grounds before the labor floor drops out. Whether we think seriously about Y as a meaning problem and not only a surveillance problem. Whether there is any credible way to construct a Z in a world with no Zenith Syndicate and no Ghem Union — and whether we start building it now, in Phase 1, while we still have leverage.

I have been living inside this problem for four volumes. I don't have clean answers. I have a formula that I think is honest about the shape of the question, and a novel that tests it as hard as I know how.

The novel is serialized here. Start at Chapter 1, "Valuation," if you want the full arc. Start at Chapter 17, "Arbitration," if you want to go directly to the scene where the formula is tested.

u/ogydugy — 12 days ago