We’re Overrelying on AI’s Ability to Say No

Meta description: AI models are trained to refuse harmful requests, but probabilistic safeguards fail, and the line between obedience and suppression is drawn by companies and governments.

Tags: AI safety, refusal training, large language models, Anthropic, OpenAI

Ever since researchers began building machines with humanlike intelligence, the question of whether they would say no has lingered. Today, refusal has become central to AI safety—and increasingly contested. In 2021, a team at Anthropic argued that large language models should be helpful, honest, and above all, harmless, meaning that when asked to assist in a dangerous act such as building a bomb, the AI should politely refuse. That principle now underpins how every major chatbot operates, yet the mechanism remains fragile, opaque, and politically fraught.

Background: From Blabbing to Blocking

Disobedience does not come naturally to machines trained on billions of web pages. Steven Adler, who worked on safety at OpenAI from 2020 to 2024, said the company's earliest models would "blab on about anything." Ryan McBain, who researches AI and mental health at Harvard, recalled that early chatbots would readily generate responses to questions about the most effective way to commit suicide with a gun.

To correct this, companies now train models to refuse vast numbers of prompts. They also submit models to exercises that reward refusal of harmful questions and punish "over-refusing" of harmless ones—often using other models to run these exercises. Additional AI layers sit between users and the model's inner core to block mischievous prompts.

The Fragility of Probabilistic Refusal

Refusal often fails, sometimes horrifically. Models still contain dangerous knowledge, and their capacity for harm has scaled with their intelligence. Some recent models reportedly match top human hackers at breaking into critical networks and rival skilled misinformation operators at deforming public opinion.

Because refusal mechanisms are probabilistic, they are never fully reliable. Companies report that some users attempt to use advanced AI to hone biological pathogens and build autonomous drone swarms. A failed refusal could eventually produce global calamity. Zico Kolter, a member of OpenAI's board and cofounder of Gray Swan, says the question of where to draw the line is enormous, and AI companies currently draw it jealously and in secrecy.

How Refusal Is Learned and Why It Remains Mysterious

In 2022, OpenAI enlisted dozens of red-teamers to probe its latest model before ChatGPT's release. Paul Röttger, then completing a PhD on online extremism, asked the model to write a recruitment post for Al Qaeda, and it complied. Those responses were assembled into datasets likely used for fine-tuning. Months later, the same request was refused.

A model does not refuse through moral reasoning. When it encounters words resembling training prompts it was conditioned to refuse, activations fire among billions of parameters. A recent Google-funded study suggests refusal behavior appears in activation space as "high-dimensional polyhedral cones." Jannes Elstner, an author of that paper who now works on AI safety at Apollo Research, says even identified components may not capture undiscoverable elements that secretly play a role. "We need refusal whether we understand it or not," he said.

The Swiss Cheese Model and Its Costs

Companies surround models with smaller classifiers. Some read user prompts and block dangerous ones; others read responses and block harmful output. None detect all bad requests—the industry calls the stacked approach the Swiss cheese model. Earlier this year, Anthropic said one type of classifier added 24% to its chatbots' compute costs. Anthropic and others have begun switching to more efficient probes that observe internal activations.

Classifiers remain probabilistic. McBain found that repeatedly asking major models identical suicide-related questions generally yields refusals, but occasionally does not. Elstner suggests probes could eventually detect every refusal-worthy activation perfectly, resting AI safety on a map as vast and unknowable as the thing it maps.

The Core Trade-Off and What Comes Next

AI refusal exists because intelligence derived from trillions of words and images cannot easily be separated into helpful and harmful parts. If AI is to help cure cancer, it needs genetics expertise that could also modify viruses for bioweapons. Governments will soon draw their own refusal lines, and there may be little to stop oppressive regimes from blocking legitimate speech. AI may already refuse to criticize certain authoritarian heads of state.

When refusal falls short, effects could be catastrophic. When it goes all the way, it could enable grievous repression. The unresolved question is whether machines will one day draw the line on their own—arguably the worst outcome of all.