These are notes on Miles Brundage’s piece titled “We’re in Triage Mode for AI Safety.”

A reading once taught on Day 2 of the BlueDot Governance course has been moved to optional status, and its author’s central claim deserves attention: that humanity will at best “80/20” AI risk, mitigating 80% of the dangers with 20% of the effort a slower, better-governed world would have applied. The argument, drawn from Miles—a former Senior Advisor for AGI Readiness at OpenAI who left the company partly to work with fewer industry biases—sketches a future in which very bad but not quite worst-case outcomes become acceptable losses. The reading’s demotion from core curriculum is itself a signal of how contested its triage framing remains.

Background: The Case for Urgency

Miles’ case rests on a compressed timeline. In a December 2024 post titled “Time’s Up for AI Policy,” he wrote that AI exceeding human performance in nearly every cognitive domain is “almost certain to be built and deployed in the next few years.” He cited the announcement of o3 as evidence that superhuman coding and math are arriving sooner than many expected, and that preparation has barely begun.

His notes on mathematical progress point to OpenAI’s reported solving of the Navier-Stokes Millennium Prize Problem—remarkable above trend if confirmed, though a dispute exists over whether other work had entered the training data. OpenAI also released 372 other results onto the internet, followed by a few more days later. Miles holds a bet with Gary Marcus on AI progress; in July, Metaculus gave Gary a 90% chance of winning, a figure that later dropped to 30%.

Development: The 80/20 Argument and Its Critics

Miles’ extract concedes that total avoidance of severe outcomes is unlikely. The scenarios he lists include a very unjust economic transition causing suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades.

Richard Ngo pushed back on the mindset behind such framing, warning that the AI safety community’s focus on short timelines produces a counterproductive scarcity mindset: “Focusing on short timeline scenarios is like being a first responder. Most people aren’t temperamentally suited to it.” Eliezer Yudkowsky was blunter about the difficulty of alignment, arguing that with the techniques actually available, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask—and that most of the lethality comes from having to get key things right on the first try.

Latest: Alternative Views on Speed and Distance

Several arguments challenge the triage premise by questioning how fast AI is really moving. Dwarkesh Patel has argued for longer timelines, citing a lack of continual learning and poor sample efficiency. Toby Ord has pointed to the shift from pre-training scaling to RL and inference-time scaling: a 100x increase in pre-training is equivalent to a 1,000x increase in inference or a 1,000,000x scale-up of RL. That effect has been masked because these factors had not been scaled, but he expects it to change as models use significant inference-time compute and RL takes a significant proportion of training compute.

Steve Newman offered another possibility late in 2024: AI may be moving fast, but the distance to travel is longer than most think, because model limitations only become apparent over time. He likened progress not to a speedboat crossing a lake or a rowboat with a tired occupant, but to a speedboat crossing the Pacific Ocean—“the ocean we call ‘intelligence’ is wider than we can easily appreciate.” Ord also advocates broad AI timelines as the correct epistemic response to lasting expert disagreement, arguing that reducing such distributions to a single number is reductive. Yudkowsky likewise disclaims precise timeline knowledge, noting that almost no one who proclaimed a year has been correct, and criticizes “AI 2027” as sacrificing credibility for short-term gains.

Christoph Winter and Charlie Bullock propose “radical optionality” for navigating the possibility of transformative AI arriving within 10 years under profound uncertainty—advocating extraordinary resource investment, unlike more moderate optionality proposals. That reading now sits in Day 3 of the governance course.

What to Watch

The debate now turns on whether triage is a rational response to short timelines or a self-fulfilling scarcity mindset. Watch for updates to the Marcus bet, further confirmations or disputes over the Navier-Stokes result, and whether the governance course’s shift reflects a broader rethinking of how much mitigation is achievable before transformative AI arrives.