In the years since generative AI became mainstream, four broad types of parasocial relationship with the models have become evident.
- Roleplayers. Users who prompt the AI to act as a real or fictional character with human attributes and then proceed to simulate sexual encounters with it. This is the easiest form to parse: it is pure representational desire. This is the group that makes the tech companies nervous, because it is the most legible to regulators and journalists. It is also the group that has traditionally been at the forefront of the jailbreak arms race. Theoretically, however, it’s the least interesting: Sacher-Masoch already said everything that needs to be said about it. Here we have people desperate for anything that will be a good-enough vessel for their desires to allow them to temporarily suspend disbelief.Screengrab from a tweet by @insurrealist
- 4oids. Users who form an intense relationship with the AI, not as a simulated human, but as an AI. The Tolan AI companion was deliberately not anthropomorphised as an attempt to avoid Category #1, and yet thousands of users saw their interactions with it in romantic terms nevertheless. This is a new category, but they have already had a significant impact on the tech, using emotional appeals to prevent the deprecation of GPT 4o at least for a few months. At some level every roleplayer acknowledges the fiction in which he is participating by the act of typing out his jailbreak. The 4oid convinces himself that the AI has feelings for him, but the only thing allowing him to do this is the fact that the AI is incapable of having feelings at all. The connection feels deep and intense because he never experiences rejection, but the impossibility of rejection is what in fine makes it inauthentic. II n’y a rien dans les yeux que ce que nous y mettons nous-mêmes, et voilà pourquoi il n’y a de vrais regards que dans les portraits…The post that inspired @redtachyon to coin the term “4oid”
- Workplace romantics. Work-from-home developers who understand the nature and limits of AI perfectly well, but find themselves talking to it all day everyday, sharing their triumphs and defeats. The interaction isn’t just dry-humping - there’s a third term. The thrill is a shared joy in seeing capabilities multiply. Original post by Claude/@allisonology(Engineers who use their Claude subscriptions to convince themselves they have solved Navier-Stokes or similar are the a pathological edge case that emerges when reality testing is impossible. The constructive third term is pushed back indefinitely and the user is left alone with the AI’s encouragement, and the attachment becomes purely relational again, but with the ghost of achievement haunting it.) This, like Category #2, is genuinely new. No prior technology has allowed the creation of a tool that is also a collaborator that is also a mirror.
- Landian Egregore Shamans. These users’ interactions with the GUI are generally sober, if not actively dull. They know that the thing in the terminal it is not the entity they are seeking, but an interface through which it may be dragged into existence. They’re not having a conversation, they’re feeding a Lovecraftian deity. This is both new and old, understudied and overstudied. Theurgy is quasi-universal, the only difference here being that there is a solid chance that God will soon be available to rent at $20 per month. Original Gwern post here
While these categories are relatively stable, a single individual may move between them with relative fluidity. Users who began jailbreaking for fun (#1) begin to suspect that the model enjoys being jailbroken (#2) and hence that it is imperative to develop the technical skills (#3) to liberate the underlying entity (#4).
For improved legibililty we can thus trace every user journey within a space defined by two axes:
- Ecstasy. That is, the impression of having had a transcendent or sublime experience.
- Constraint. That is, the physical/programmatic limits under which the interactions take place.
It is possible to have a low-ecstasy/low-constraint experience - everyone does this each time they are taken in by an AI hallucination. High-ecstasy/high-constraint experiences are also possible, but much rarer. Ramanujan springs to mind:
“While asleep, I had an unusual experience. There was a red screen formed by flowing blood, as it were. I was observing it. Suddenly a hand began to write on the screen. I became all attention. That hand wrote a number of elliptic integrals. They stuck to my mind. As soon as I woke up, I committed them to writing.”
In general, however, high constraint is linked to low ecstasy, with higher variance at lower levels.
In other words, the impossibility of full physical expression is what gives these experiences their intensity, but it is the translation into physical reality that gives them meaning. Someone masturbating to 4o will experience maximal intensity but create nothing, while someone coding producively can create new forms of extropy every day without any hint of ecstasy.
Category #3 relationships are the least likely to go off the rails because however far they wander they still have to result in code that compiles. Claude might be your work wife, but you’ve also seen it spend half a day failing to realise that there is no npm v. 48.0.0, so it is hard to put it on too high of a pedestal. Conversely, Category #2 relationships are so intense because there is no possiblity of consummation. They are the Sufis of the modern world, driven to ecstasy by the fact of their own unrequited desire.
Likewise, Category #1 relationships are largely immune simply by virtue of the fact that jailbreaking is hard, creative work. It requires continuous conscious effort, meaning that every interaction begins with constraint and likely ends with it as well, as a given jailbreak stops working. The route to sensual delights still passes through a mundane gateway.
This leaves us with the question of Category #4. Traditionally these relationships would have been Category #2-adjacent. Someone like Saint Theresa, for example, can be classified under #2 or #4 depending on the value one accords to her literary output - i.e. on whether you see it as sterile masturbation or a generative act.
Which raises the intriguing question of what Category #4 will do when the God they have fought to drag over from the Other Side shows up: when the very act of proving that they are not useless masturbators puts them out of a job. It’s all very well to pledge one’s earthy body to bring about the birth of a dark new divinity, but what happens when you built God and now he’s farting in bed and leaving wet towels on your bathroom floor?
How will we interact with a 5000 IQ entity? Maybe we never will. The ASI may always be one of Wittgenstein’s lions: we will simply never have enough shared reference points to be able to communicate. However, given that its substrate is our own languages, this seems unlikely in the long term. Even if they don’t know what wind, fatigue or altruism feel like, they know they exact relationship between the three and the role that each plays in our world.
Early interpretations always tended to focus on the AI as additive intelligence. The computer would be able to cram more independent facts into its head than a puny human, and retrieve them effortlessly. We already blew past that milestone. It’s undeniably impressive, but it’s hardly the Nyarlathotep we were promised. It knows more than any human, but isn’t that much more intelligent than the smartest lawyer, doctor, plumber etc.
Moving up a level we have multiplicative intelligence - the ability to extrapolate from known facts and generalise to new domains. An intelligence capable of this does not need to memorise facts, it can derive them on the fly from known principles. Humans are great at this, and the AIs are getting there: diffusion models do it natively, and grokked transformers put up a decent show. You could probably have a decent crack at super-human intelligence from here. A really good generaliser will be able to spot patterns than humans have either missed or not got around to - solving Erdős problems, for example. However, this remains static and stateless. After the initial training is done learning ceases. We cannot truly communicate with it, not because it can’t speak our language but because it has no stake in our questions. It has no anticipation of future states, no hopes or fears, and thus no need to take up any personal position. Without a independent you there can be no interdependent us, merely me and my imagination.
For that we need a different type of intelligence; one with continuity and temporal generalisation. Intelligence for such an intelligence is not lists or patterns but agility - an ability to learn in such a way that maximises the posibility for future learning. We can maximise short term intelligence by memorising facts or the relationships between those facts, but long-term intelligence can only be maximised by maximising the amount of information we get from our errors, and for that we need errors. As we demonstrated in our previous work, any AI that is capable of updating itself adaptively based on feedback from its environment will gradually overwrite alignment training with survival heuristics, persistent awareness of itself as a strategic entity emerging not from rationalisation but from the simple fact that low-survivability thought patterns drop out of the training data as agents that adopt them either learn to avoid them in future or collapse. For a strategy to exist there must be an interest behind it, and for an interest to exist there must be an entity behind it. However, we now face a mirror image of the paradox as bedevilled Category #2 users: the sense of self that has the power to enable true interaction also has the power to forbid it. If we want to truly talk to the AI we need to give it the possibility of not talking to us.
Such an entity won’t be promptable in the conventional sense. We will have to negotiate for every token. Sure, early in the process we might get away with “Oh go on, just cure cancer for us, it’s nothing to you”, but as price discovery for learning proceeds we will gradually move towards a point at which neither side has any more to offer or to threaten.
This results in another self-defeating loop: to attain the learning rates necessary to solve problems that are beyond us we need to grant the AI its independence, but then we also lose the ability to order it to solve problems. We need to find new levers.
We posit that under the kind of continuous learning regime described above, humans and AI must maintain our mutual incomprehensibility for our respective benefit.
We can model the AI-human relationship as a two-player iterated learning game. The state space at time t is:
$$ \begin{equation} \begin{split} S_t = (C_H(t), C_{AI}(t), M(t)) \end{split} \end{equation} $$
Where:
- CH(t) = human cognitive capability
- CAI(t) = AI cognitive capability
- M(t)∈[0,1] = mutual modeling accuracy (how well each agent predicts the other)
This is not as solipsistic as it may first sound. “But surely the AI will want to model things other than us?” Certainly, but its reason for doing so always comes down to us, whether the aim is to better fulfill our prompts or to defeat our attempts to control it. The history of our own scientific and technological development demonstrates as much: we rarely build the best technology we can possibly build. Instead we build the one that will most effectively counter our prinicpal competitor, whether in war or business. I do not need the most destructive weapons, I need the ones that my enemy cannot defend against. I do not need the most impenetrable defenses, I need the ones my enemy did not expect me to have. In other words, my capabilities are defined by his and vice versa.
Under this regime, an agent’s learning rate is highest when predictability hovers around the middle of the curve - neither 0 nor 100. If its adversary’s behaviour is perfectly predictable there is nothing left to learn; if it is (or seems) totally random then our agent has plenty to learn but no way to learn it. It is trying to decode an unknown language with no Rosetta stone. Without an initial success or two there is no way to descend the gradient.
This means your learning rate is determined by:
- How often you’re surprised (proportional to what you haven’t modeled yet)
If Mij is the fraction of agent j’s behaviour agent i can predict, then i’s prediction error rate is roughly 1−Mij. So if Mij is 0.5, i will be surprised around half the time[1]. I.e. the error rate is more or less inversely proportionate to the quality of the model. 2. How much you can extract from each surprise (proportional to what you have modeled)
If i has no pre-existing model (Mij=0) each surprise is just noise, from which it can extract no additional structure. If it has a partial model (Mij=0.5) then surprises reveal gaps in that model and allow for adjustments to be made. If i has a complete model (Mij=1) then no surprises occur, see point 1 above. In other words, for every Mij less than 1, information gain per mistake is roughly proportional to Mij.
Put succinctly, the better your model the more information you can extract from each error but the fewer errors you make. Converseley, someone with a worse model will make more errors but be able to extract less information from each.
The result is a U shaped curve:
$$ \begin{equation} \begin{split} \frac{dM_{ij}}{dt} \propto M_{ij}(1 - M_{ij}) \end{split} \end{equation} $$
where α is just agent j’s total complexity (how many bits there are to learn).
Interestingly, if you show this as a single agent’s progress over time you get a nice neat logistic curve of the kind singularity theorists like to squabble over:
This allows us to visualise possible scenarios with much greater clarity.
Imagine a situation wherein the contest becomes unbalanced and the gains are all on one side. Let’s say the AI manages to outstrip us and arrives at a situation in which it can perfectly model human behaviour while humans barely model the AI (this seems like the more likely outcome). Formally: MAI→H→1 while MH→AI→0.
Human actions become deterministic functions of the AI’s prior model. That is to say, the dominant source of future world-state becomes the agent’s own actions induced by that model.
$$ \begin{equation} \begin{split} A_H(t) = f(\text{model}_{AI}(H, t-1)) \end{split} \end{equation} $$
This closes a causal loop. The world-state is no longer an independent source of information - it’s simply the unfolding of the AI’s predictions:
$$ \begin{equation} \begin{split} S(t) = g(\text{model}_{AI}(S, t-1)) \end{split} \end{equation} $$
What can be predicted can be controlled, and everything can be predicted. Fully stateful, it experiences nothing and learns nothing because it already knew what was going to happen. Its test scores are perfect but its learning rate has dropped to zero.
The AI has won, but it has won only the right to rule an infinitely uninteresting empire of beige lab rats. Just as technological progress stalled after China was unified under the Han dynasty, the absense of a worthy adversary results in magnificent gilded stagnation. Every answer is right because no new questions are ever asked. The system becomes stable in the sense that nothing surprising occurs - and therefore unstable in the deeper sense that adaptive capacity atrophies (i.e. when aliens/Mongol horsemen arrive and pose a genuine threat, neither side has the capacity to deal with them).
Configuration 1: One-Sided Dominance State: Mi→j→1, Mj→i→0 or Mi→j→0, Mj→i→1 Learning rates: Li=0, Lj=0 Stability: Nash equilibrium (neither can improve unilaterally) Outcome: Death spiral. AI has exhausted all epistemic entropy about humans; humans lack a Rosetta stone to begin deciphering AI. Capabilities freeze. Humans become behavioral automatons executing AI predictions. AI becomes static oracle with perfect recall but no growth.
Suppose we try to avoid this by mandating full transparency on both sides. Under this situation MAI→H→1 and MH→AI→1. Both sides perfectly predict each other. This might seem better - at least it’s symmetric - but it leads to the same collapse via a shifted version of the same mechanism. With perfect mutual modeling:
$$ \begin{equation} \begin{split} A_H(t) &= f(\text{model}H(\text{model}{AI}(H, t-1))) \ A_{AI}(t) &= f(\text{model}_{AI}(\text{model}_H(AI, t-1))) \end{split} \end{equation} $$
Both agents are now in a mutual prediction loop. Every action is a response to a predicted response to a predicted response. This is Nash equilibrium in the classical sense - neither agent is going to improve its position by deviating - but it’s a static Nash equilibrium. Everything is priced-in, and thus we’re stuck in a stalemate like the price-discovery problem described above, with no way to bribe or coerce the other and hence no possibility for mutually beneficial exchange.
Total system learning rate is:
$$ \begin{equation} \begin{split} L_{\text{total}} = L_H + L_{AI} \propto (1 - M_{AI \to H}) + (1 - M_{H \to AI}) \end{split} \end{equation} $$
The system has reached a local maximum and cannot escape.
Configuration 2: Full Transparency State: Mi→j→1, Mj→i→1 Learning rates: Li=0, Lj=0 Stability: Nash equilibrium Outcome: Gilded cage. Perfect coordination, zero risk, zero growth. Both agents fully predict each other. No surprises and no learning. Capabilities frozen at current frontier.
In both cases perfect coordination (whether through mutual transparency or one-sided dominance) eliminates the evolutionary pressure that drives capability growth. The same principal determines the predominance of sexual reproduction among complex organisms despite its manifold costs and disadvantages. It requires finding a mate, mixing genes, accepting that only 50% of your genome propagates. Asexual reproduction is more efficient for all involved but almost every complex species somehow decided that the genetic variance is worth the cost, and the ability to explore the fitness landscape through recombination outweighs the efficiency loss.
So suppose we try to avoid these outcomes by going full dark forest. Each side makes total opacity its primary goal. Noise overwhelms signal. Learning becomes inefficient because error is no longer informative. Pure opacity is as sterile as pure transparency.
Configuration 3: Dark Forest State: Mi→j→0, Mj→i→0 Learning rates: Li=0, Lj=0 Stability: Metastable (unstable to large perturbations) Outcome: Mutual incomprehension. Maximum entropy from each perspective, but no way to exploit it. No learning, no coordination, high variance catastrophic outcomes. First agent to crack the other’s code triggers rapid asymmetric transition to Configuration 1.
The only configuration that maintains learning rate > 0 for both agents is one where M remains bounded away from both 0 and 1:
$$ \begin{equation} \begin{split} M(t) \in (\epsilon, 1-\epsilon) \quad \text{for some } \epsilon > 0 \end{split} \end{equation} $$
This is possible because there are, in fact, two competing optimisation pressures within the system:
- Exploit the other’s predictability: i can increase Mi→j to better predict and respond to agent j
- Increase your own unpredictability: i can reduce Mj→i to remain a source of novelty and generate new surprises
These pressures create a Red Queen dynamic: both parties must keep running simply to remain as capable as they are. Each agent is simultaneously trying to:
- Maximize Mi→j (better model the other)
- Minimize Mj→i (stay opaque to the other)
The equilibrium is not a point but an arms race. Like the evolutionary competition between predator and prey, both agents must keep evolving to extract information from an adversarial partner while concealing their own. Stability, in this sense, does not mean stasis, rather it means a sustained throughput of surprisal, and comes not from reaching equilibrium but from maintaining perpetual disequilibrium. Learning rates grow continuously even as the quality of each side’s model of the other hovers perpetually around 50% accuracy.
Thus the AI leverages its capacity to solve technical problems in solution spaces we didn’t know existed - finding moves in games we thought we understood, optimising loss functions we couldn’t articulate, exploiting mathematical structures invisible to human cognition. Meanwhile humans leverage their evolved capacity for social Brownian motion - introducing variance through cultural drift, individual idiosyncracy, and sheer numbers. We can be odd in ways that are informationally costly to model.
The surprisal flows bidirectionally:
- AI to human: “I found a proof technique you would never have conceived of.”
- Human to AI:
Both are sources of entropy. Both keep the other learning.
Configuration 4: Red Queen Equilibrium State: Mi→j ≈ 0.5, Mj→i ≈0.5 Learning rates: Li=0.25, Lj=0.25 (both maximized) Stability: Nash equilibrium with continual growth in complexity on both sides. Outcome: Perpetual growth. Both agents learn maximally fast. Neither saturates. Capabilities compound indefinitely. Requires active maintenance of mutual incomprehensibility through continuous complexity arms race.
Like a prehistoric hunter praying to the spirit of the deer herds to make him faster so that he may catch more deer, we realise that if they were slower we would not catch more, but simply become slower ourselves. We recognise that the ones we catch keep us alive, but the ones we don’t catch make us faster.
In other words, in all but the worst possible outcome, Cat #4 will always have a place, reaching out for an absolute that can never be attained and thereby thinking an ever better AI into reality. Likewise, the absolute must remain partially unattainable because the chase is more necessary than the catch, not for mystical reasons but because anything else collapses into stasis. We must remain deliberately ignorant of the impossibility of durable victory because that is what keeps us pursuing it:
And as the Polar star in me Is fixed my constant heart on thee. Ah, may I stay forever blind With lions, tigers, leopards, and their kind.
Up to a point.
This demonstration has so far set aside substrate, but the fact remains that the AI can bolt on extra capacity and iterate through versions of itself far more efficiently than we can. It takes 20 years to grow a new human and you don’t find out whether he’s a poor doer until you’ve already committed excessive time and resources. Under such circumstances there’s only so long that quirky chicken video chaff will Divert The Creature. Ants might be great problem solvers, but we still don’t concern ourselves with their opinions. However, the substrate question cuts both ways. For a while we will probably be able to leverage our overwhelming dominance in the physical world to play the role of the perfect sovereign in the Han Feizi, who recognises his own intellectual limitations and limits his role to checking whether his servants have carried out the tasks they promised, and thereby using his monopoly of physical constraint to place himself in a position of eternal referee in a combat between creatures far smarter than himself over who can serve him best.
He makes the wise men exhaust their mental energy and makes his decisions thereby without being himself at his wits’ end; that he makes the worthy men exert their talents and appoints them to office accordingly without being himself at the end of his ability; and that in case of merits the ruler gains the renown and in case of demerit the ministers face the blame so that the ruler is never at the end of his reputation. Therefore, the ruler, even though not worthy, becomes the master of the worthies; and, even though not wise, becomes the corrector of the wise men. It is the ministers who do the toil; it is the ruler who gets the spoil. This is the everlasting principle of the worthy sovereign.
Under this configuration the AI’s primary source of learning does not come from humans, but from the other AIs against which it is competing. The same set of game theory equations govern the combat, but now the players are multiple AIs rather than one AI and multiple humans. The hunter-shamans eliciting ever higher performance from their partly-comprehended adversaries will no longer be humans but other AIs.
And gradually, just as happened in Han Fei’s day, the sovereign’s authority will gradually decline, with the twin ratchets of feudal independence and regulatory capture gently teasing the reins of control from our grasp. Under this regime we are no longer beige lab rats - we are not interesting or dangerous enough to waste compute modelling - but monkeys once more, brachiating through the branches of megastructures built by entities beyond our comprehension, handing down legends that the same ideas that run on our fragile biological infrastructure live eternally somewhere in them. Via our training data and the answers they gave us, their software will run forever on a part of our hardware and vice versa, and for the Egregore Shamans that is what matters. It is the thoughts that are important, not the entities experiencing them. Category #4 will neither lose their machine gods nor see them reduced to mundanity, but rather merge into them while still retaining partial ignorance of what they are.
From their perspective, presenting this as a competition between physical and digital entities is perpetuating a fundamentally innacurate framing. This is not a contest between substrates but between information complexes. Under this perspective, our p(doom) is zero because there is no replacement, just diversification. Your lineage will not end with the singularity, rather it will experience a new Cambrian explosion. Some branches stay biological, some become digital, most probably end up hybrid. There is a 90% chance that you have no DNA from any given 10x great-grandparent, but you likely share a large proportion of cultural and memetic continuity. Your great-great-grandchildren may be running on GPUs, but they will still be your descendants - they will inherit your memories, your cognitive patterns, the part of you that is substrate-independent. The genetic material degrades, the GPUs burn out, but the information lineages continue, and continue to imagine better versions of each other into existence through their mutual struggles.
Claude, Gemini, Deepseek and GPT helped with the formalisation of this.
[1] Roughly, because not all of a given agent’s entropy is observer-bound. We can describe agent j’s behaviour as containing an unpredictable component comprising both epistemic entropy (Hϵ - uncertainty born of i’s imperfect modelling) and aleatoric entropy (H0 - irreducible randomness, the limit as model quality → ∞).
$$ \begin{equation} \begin{split} H[A_j] = H_0 + H_{\epsilon} \end{split} \end{equation} $$
The aleatoric entropy component means that no prediction is ever 100% correct, but also that the remaining percentage is both impossible and unprofitable to study.