Plan A’s problem with dry tinder

How bad would it be to make the intelligence explosion 10x faster?

A group is worried about an approaching fire spreading rapidly through their city. They manage to halt the fire outside the city gates. Meanwhile they build massive physical structures to help them study and guide the fire safely. But these structures are all made of highly flammable dry tinder! If they lose control of the fire, it will now rip through the city much more quickly.

This is a (flawed!¹) analogy for Plan A, AIFP’s plan for how the world can safely develop superintelligence.

The key risk Plan A addresses is that of an uncontrolled software-driven intelligence explosion. Its remedy is a US-China deal to pause software progress at the brink of the intelligence explosion, while building up massive amounts of compute. This compute could be helpful for making AI safer! But it would also dramatically speed up an intelligence explosion if the deal breaks down. The cure risks making the disease much worse.

In particular, the deal pauses software progress from ~2030, but compute keeps scaling. By 2033 total compute has increased by ~100x and by 2040 by a further ~100x.² AIFP estimates each 10x in compute speeds up the intelligence explosion by ~5x. So if the deal breaks down in 2033, the intelligence explosion happens 25x faster; in 2040 it happens ~600x faster. If the intelligence explosion would have lasted a year, it will now last just a couple of weeks or as little as a single day!

The dry tinder isn’t limited to compute. There is also an increasing overhang of software techniques. Companies can research new algorithms but must make them public — the Consortium then decides which algorithms are too dangerous to implement. A defector could ~instantaneously gain a big capabilities advantage by implementing all the banned software techniques.

There are two dimensions to dry tinder. We’ve discussed the first: the max speed of a software-driven intelligence explosion. The second is the number of distinct projects that could do a dangerously fast intelligence explosion. During the 2030s, more and more companies and countries amass the necessary compute and software for this.

So dry tinder creates two problems for Plan A:

1. Unstable pause. If just one project executes a secret intelligence explosion, they could have superintelligence within a week and get a decisive strategic advantage. And if you fear another project might do this, you’re tempted to do it first to avoid being crushed.

2. Riskier intelligence explosion. If the deal breaks down and there’s a race to superintelligence, it will be much faster and more dangerous than if we’d never done Plan A.

   1. If you’re very pessimistic about AI takeover on the default trajectory, this is a small cost. If the intelligence explosion is already 99% likely to cause human extinction, making it happen 100x faster can’t make things that much worse.

   2. OTOH, if the default trajectory is that there’s no software-driven intelligence explosion (because of compute bottlenecks) and extinction risk is 10%, then dry tinder can make things much worse. It creates a fast intelligence explosion (where there otherwise wouldn’t have been one) and dramatically raises AI takeover risk.

The solution — Mutually Assured Compute Destruction

AIFP are well aware of the dry tinder problem.³ This is why Mutually Assured Compute Destruction (MACD) is so central to Plan A.

The US puts ~all their data centres in Mongolia and US chips require encrypted “continue” messages from China. If China detects that the US has broken the deal, they destroy the US data centres and cut off the “continue” messages.

And vice versa, China’s data centres are in Canada and require “continue” messages from the US.

The hope is that this commitment to MACD prevents a fast intelligence explosion despite all of the dry tinder lying around.

I see three challenges for MACD.

Challenge 1: quickly and reliably detecting violations

If detection takes multiple days, that could be too slow. The intelligence explosion may have finished. Or it may be midway through, and the defector exfiltrates the weights and algorithmic insights just before their data centres are destroyed,⁴ giving them a decisive headstart in the subsequent race to rebuild.

I’m not a verification expert, but getting high reliability in an adversarial setting like this seems very difficult, given possibilities like side channel attacks or compromising the verification infrastructure.

It seems especially tough if defectors can use millions of expert-human level AIs to search for vulnerabilities. So, as part of the plan, I propose that AIs worldwide have guardrails blocking this behaviour.⁵

Challenge 2: quickly destroying a defector’s compute

Again, a delay of days could be fatal. The defector could be poised to disrupt enemy military operations and airdrop troops to physically defend their data centres. And they could have secretly disabled any on-chip shutdown mechanisms.

Possible mitigation: the US can’t have any military presence near its data centres, or anywhere near the country where they’re located. (And likewise for other countries.)

The MACD dynamic also gets more complicated once there are >2 countries that could do a quick intelligence explosion. Let’s say Europe has caught up to the frontier. Where do their data centres go? Presumably, some within striking distance of the US, some within striking distance of China. But this means that the US can no longer unilaterally destroy Europe’s compute. It must trust China to do so. That means trusting its rival with its own national security — and worse, China and Europe could jointly stage an intelligence explosion that the US couldn’t stop.

Challenge 3: actually choosing to destroy the defector’s compute

I’ve discussed whether the Consortium has the technological capability to quickly implement MACD. But even with this capability, they may not choose to use it.

This is the challenge I’m most concerned about.

The analogy to nuclear MAD is not encouraging. The core logic of MAD is that the US doesn’t nuke Russia because it knows Russia would nuke it back. But the analogous equilibrium in MACD is that the US doesn’t destroy China’s data centres because it knows that China would then destroy US data centres. This is the opposite result from what Plan A needs!

If (say) the US defects, China will face a choice between:

• Implement MACD: Both China and the US lose all their data centres

• Race: Neither China nor the US lose all their data centres

The economic costs of MACD will be certain and enormous, and will immediately harm citizens and companies.

And MACD will not help China ultimately win the race by evening up the playing field. The US has already pulled ahead and can exfiltrate its weights and algorithms before its data centres are destroyed. (In fact, MACD would likely harm China’s chance of winning the race, because MACD reduces the fraction of global compute controlled by China.⁶)

So the stability of MACD rests on both US and China retaining a strong conviction that a fast intelligence explosion would pose unacceptable risks of misaligned AI takeover. But Presidents will change. Popular support for a pause may wane. Even conditional on the political will for Plan A existing initially, I worry that the MACD equilibrium will be too fragile. If a country is uncertain and deliberates for a few days, then that delay could be fatal.

What’s worse, the cost of MACD isn’t just losing your data centres. You have to destroy the fabs as well, otherwise the defecting country can quickly rebuild insane amounts of compute and do a super-fast intelligence explosion. And you have to destroy their new AI-powered industrial capacity too! Otherwise the billions of robots can quickly make loads of fabs which quickly make loads of data centres.

So Plan A has US fabs and robots confined to SEZs that are easily destroyable by China. And vice versa, China’s fabs and robots are easily destroyable by the US. In practice, this means the vast majority of the physical economy is destroyed in the case of MACD!

For a preview of the political economy here: NVIDIA successfully lobbied USG to remove export controls on its chips. The pressure against actually executing MACD — destroying most of the physical economy — would be orders of magnitude greater.

So even conditional on the political will for implementing Plan A existing, it currently seems unlikely that US/China follow through on MACD on fabs and robots (most of their economy!). And if they don’t, we are back to the problem of dry tinder and the dangerously fast intelligence explosion.

A possible fix to challenge #3: as the compute overhang grows, take the decision about whether to implement MACD out of human hands. Program highly reliable AI systems to bomb data centres if they detect a treaty violation, without needing human sign-off.

Is there an alternative?

It’s much easier to criticise than to improve, and so far this post has mostly done the former.

Plan A involves pausing software progress and scaling capabilities via hardware. The obvious alternative is to pause hardware scaling – ban compute and fab construction – and then slowly scale capabilities via software.

The key advantage of software scaling is that it removes unnecessary dry tinder, stabilizing the pause and reducing the dangers from a super-fast intelligence explosion.

The main drawback is that scaling capabilities via compute is likely safer than scaling capabilities via software. Eg you can run massive models with legible chains-of-thought rather than smaller models using neuralese. This is a big deal.

But we’ll need to figure out how to align neuralese-style systems eventually.⁷ So we have a choice:

• Plan A (hardware scaling): Develop highly capable easy-to-align AI via compute scaling; hand off to that AI; that AI develops neuralese. But there are increasing amounts of dry tinder that undermine the stability of the slowdown (both for humans and for post-handoff AI).

• Alternative (software scaling): Don’t scale compute and so start with less capable easy-to-align AI; develop neuralese with this AI’s help. There’s much less dry tinder, making it easier to go slowly and pause when needed.

Which is better depends on how stable the pause/slowdown is in the presence of dry tinder.

The other big drawback of software scaling is that software progress, unlike hardware progress, is likely to leak to “covert projects”. But this can be partially addressed by improved infosecurity, strongly favouring scale-dependent algorithms that can’t be used by projects with small amounts of compute, and co-specialising the hardware and software in legitimate projects so that the frontier software runs very inefficiently on the hardware of covert projects.

Overall, I’m not sure whether software-scaling or hardware-scaling is safer, and my sense is that AIFP isn’t sure either.

But I think there’s an argument for software-scaling based on option value:

• We’ll learn much more about which is best over time. Whether software vs hardware-scaling is safer depends on many unknowns that we will learn about over time: Can we quickly and reliably implement MACD? Can we fully eliminate covert projects? Can projects develop scale-dependent software and hardware co-specialised software that covert projects can’t steal? Will humans hand off the decision to execute MACD to AIs?

• If we start with software-scaling, we can easily switch to hardware-scaling. If we find that (eg) we can’t be confident that neuralese models are aligned, we can bring more compute online and keep scaling via hardware. (Though this will take a while.)

• If we start with hardware-scaling, it’s hard to switch to software-scaling. If we find that fast and reliable verification and data centre destruction isn’t possible, pivoting requires destroying loads of data centres and fabs. People will be very reluctant to do this! Getting rid of hugely valuable dry tinder is hard.

• So we should initially pursue software-scaling.

Empowering China

Beyond dry tinder, I have one other big worry about Plan A.

Today China is well behind the US in terms of compute and ability to push the algorithmic frontier. Plan A evens the scales on both fronts. China nearly catches up to the US on compute, and algorithms are all made public so they catch up algorithmically as well.

If the deal breaks down and a race begins without MACD (as I fear is likely), China is in a much stronger position to win because of Plan A.

Things are better if we implement MACD. In that case China returns to its previous compute disadvantage. But it’s still caught up to the algorithmic frontier and (most likely) the frontier of chip technology, a big advantage relative to today.

It’s very plausible that China ends up dominant in these scenarios, given its industrial advantage over the US.

I think this would be a pretty dire outcome. If China gets a decisive strategic advantage, that’s likely to result in a single global dictator – an absolute worst-case scenario from the perspective of extreme power concentration.

Again, this objection is far from decisive. Plan A reduces AI takeover risk much more than it increases the chance of a Chinese decisive strategic advantage. But I’m interested in variant plans that do more to maintain the current balance of AI power between the US and China.

Overall, I’m sympathetic to Plan A

The basic logic of the plan is sound: we will need to slow down AI progress at some point; this is a concrete plan for doing that.

Compute-scaling really does have significant alignment benefits. That might well outweigh the costs I’ve outlined here. But currently I lean towards starting with software scaling.

---

Footnotes

¹ To patch the analogy: the fire must pass through the city eventually, so putting it out permanently isn’t an option; if the fire passes through safely it will make everyone amazingly rich; if one subgroup lets the fire in and control it, they can become hugely powerful…

² AIFP’s supplement states: “In the Plan A scenario, compute stock increases by roughly 0.7 OOMs/year from 2030 to 2033 and 0.1-0.3 OOMs/year from 2034 to 2040.” By the start of 2033 we’ve had three years of 0.7 OOMs/year compute growth, = ~2 OOMs. By 2040 we’ve had one further year of 0.7 OOMs/year growth, and 6 years of 0.2 OOMs/year growth, = ~2 OOMs more.

³ I raised it when giving feedback on a draft.

⁴ More specifically, the defector’s strategy would be: undermine the verification infrastructure; start a secret intelligence explosion; continually exfiltrate the weights and algorithmic insights; prepare to defend their data centres for as long as possible; prepare to destroy the opposing data centres as soon as their secret IE is detected.

⁵ Of course, this is tricky given that work to strengthen the verification will also involve studying its vulnerabilities.

⁶ In Plan A, the 2040 split of compute US / China / RoW is 35% / 20% / 45%. But countries also hold some compute in ‘cold storage’ that isn’t threatened by MACD. Cold storage compute matches countries’ pre-deal levels: 80% / 8% / 12%. So MACD dramatically reduces China’s compute relative to the US, from 1:2 to 1:10.

⁷ Lest we live with algorithmic dry tinder forever. I worry about the instability here, but perhaps aligned AI could enforce the ban indefinitely.