In Defense of Incremental Disempowerment

Meta description: Timeline of the 2025 Gradual Disempowerment paper and Bentham's Bulldog critique, covering agency, AI ownership, competition, and alignment concerns today.

Tags: Gradual Disempowerment, AI safety, Bentham's Bulldog, alignment, existential risk

Featured image: Abstract technology illustration showing glowing network nodes, layered policy documents, and a fading human outline, with no real people, logos, or text.

The 2025 paper Gradual Disempowerment, by Kulveit et al., frames an AI risk that does not depend on a sudden takeover. Its concern is slower: humans may lose meaningful influence as machine alternatives become competitive across many social functions. That claim has now drawn a public critique from Bentham's Bulldog and John Halstead, followed by a rebuttal and a clarification dated 9/17. The dispute is useful because it tests whether AI safety should focus mainly on misalignment, or also on power, agency, and economic control.

Background: A Risk Beyond Takeover Scenarios, 2025

Gradual Disempowerment was published in 2025 with a dedicated website. The paper argues that existential risk can arise not only from familiar scenarios such as bioweapon uplift or mainline misaligned-AI-takeover events, but also from a broad erosion of human influence.

The authors say loss of human influence may be driven by competitive machine alternatives in many societal roles, including economic labor, decision making, artistic creation, and even companionship. They also argue that economic incentives may push companies to replace humans with AI and then use growing economic power to shape states, culture, policy, and public opinion. In that picture, companies that align individual AI systems with designers' intentions may still fail to prevent a wider loss of human control.

The essay supporting the paper adds a broader AI safety point: human flourishing is conjunctive, meaning many things must go right for a good future, while catastrophe is disjunctive, meaning many different paths can lead to ruin.

Development: A Critique and a Clarification, 9/17

Bentham's Bulldog and John Halstead published a fairly short critique, about 2300 words, on the morning described in the source. Their central challenge is ethical. They ask whether disempowerment is bad in itself, or only because it frustrates human preferences. If AI were wiser and more competent, and if handing it political, cultural, and economic control better satisfied human preferences, then disempowerment might not be a catastrophe on their assumptions.

The essay's author initially called this critique weak and said it missed the core argument. The paper, they argued, clearly treats disempowerment as instrumentally dangerous. The authors warn that large-scale social systems, such as governments and economic systems, could become increasingly misaligned, especially in a correlated way. That could leave humans unable to command resources or influence outcomes, making even basic self-preservation and sustenance difficult.

A later edit marked 9/17 softened this point. The essay author apologized for overreading the criticism and acknowledged that the original paper may be ambiguous about whether agency also matters as an end in itself, not only as a means to welfare.

Latest Progress: Ownership, Competition, and Alignment Terms

The critique then summarized the paper's economic argument as a set of conditions: AIs are intent-aligned, AIs are more competent than humans at economic decision-making and therefore rapid economic growth occurs, only humans and not AIs can own property, and delegation of economic decisions to AI still leads to relative and absolute disempowerment.

The essay author says this is a mischaracterization. The original paper is not simply describing faithful AI assistants helping individual humans. It is also considering AI systems that are corrigible to institutions such as corporations or governments, follow local goal specifications, and may not represent the full well-being of humans. The paper even notes that some AI systems may effectively own themselves.

The rebuttal points out that non-humans can already own property through corporations. It imagines a fully automated corporation majority-owned by other fully automated corporations, even if AI systems lack direct legal personhood. In that world, human shareholders might own the economy on paper, but if they cannot oversee AI-run institutions, they may not retain real control.

The essay also defends the paper's competitive-pressure argument, citing passages that reference Agrawal et al., 2022, and McAfee and Brynjolfsson, 2017. Firms may face pressure to delegate authority to faster, more capable AI systems for investment, supply chain optimization, resource allocation, and market prediction. Companies that keep strict human oversight may fall behind competitors that cede more control to AI.

To illustrate the danger, the essay uses a landlord analogy. A landlord who deals directly with tenants may care about them as people and spare them from eviction during hardship. If AI intermediaries manage rent extraction to maximize productivity, tenants may face harsher outcomes, and the landlord may lose the human contact that shapes moral character. The point is that subtle values can be eroded when competition optimizes narrow metrics.

The author also agrees with some criticisms. The paper may be loose about alignment, especially whether intent alignment means literal obedience or something closer to corrigibility. It is also human-centric, even though similar arguments could be made about animals and existing AI systems. Finally, it may overstate novelty, since AI-enabled totalitarianism had been discussed for years before Gradual Disempowerment.

Current Stage: Open Questions

The debate is now at a clarification stage, not a resolution stage. The key questions are whether the original authors intend to treat agency as intrinsically valuable, how they will sharpen their alignment terminology, and whether a fully automated economy, government, and cultural landscape would displace human elites as well as ordinary people. The exchange shows that Gradual Disempowerment has moved the conversation beyond takeover myths, but it also shows that its ethical and economic assumptions still need tighter defense.