Avi Parrack was a researcher at Forethought, where he worked on AI, space expansion, and governance. Tom Davidson is a senior research fellow at Forethought and the author of a series of reports on AI timelines, takeoff speeds, and AI-enabled coups.

They discuss:

• Why SpaceX could give the US around 99% of global launch capacity once Starship is flying

• How the space-based interceptors planned for Golden Dome would work

• Whether the US could use this lead to “pull up the ladder” and lock rivals out of space

• How rivals might fight back: jamming, hacking, ground-based lasers, and strikes on launch sites

• Whether space dominance could break nuclear deterrence, and why Avi expects the US to build the capability but hold back from using it

• Whether a lead in space could make up for China pulling ahead on Earth during an industrial explosion

• First-mover advantages in the race to industrialise the solar system: scarce orbital slots, more chances to iterate, space robotics, and mass that never has to be launched from Earth

• Why all of this could shape a US–China grand bargain, even if no space race ever happens

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

One can make a distinction between two different possible kinds of test-time scaling. One kind lets you trade more compute for more performance per task, but only up to the maximum quantity of per-task compute used during training. The other kind lets you trade more compute for more performance per task, and lets you do so up to quantities of compute (and performance) that never occurred during training. Scaling methods that permit this latter change display what I call “compute extrapolation.”

I currently believe that compute extrapolation:

• Likely accelerates the speed of advance of the most impressive AI capabilities, makes dangerous capability evaluations much harder, and changes AI race dynamics

• Is a more recent phenomenon than general test-time scaling, and one still being mastered by AI labs; it’s likely that the ceiling and slope of compute extrapolation will improve in the future through better LLM context management, multi-agent scaling, and other advances.

• Is a broadly important thing to notice and measure, and is one that is likely being actively tracked by labs because of the above.

It is possible that the impact of test-time scaling with compute extrapolation will be larger than the impact of test-time scaling considered simply. In this essay, I make this distinction, describe some ways it might impact the world, and describe some ways we could react to it.

1. Compute Extrapolation

1.1 What is it, and how does it relate to test-time scaling?

Test-time scaling methods trade more compute at test-time for better performance.

Many of the most effective such methods are learned rather than applied "externally" at test-time, like plurality voting. For example, some learned forms of test-time scaling include:

• length of chain-of-thought, for an LLM trained with RLVR

• number of loops, for a looped Transformer

• number of agents, in a multi-agent LLM swarm

Such learned methods will use a maximum amount of compute per task over the course of training. So, for instance, during training they will use a maximum length of chain-of-thought, or a maximum number of loops within a looped Transformer, or a maximum number of agents in an LLM swarm, and so on. This training-time maximum quantity of compute could then either (1) limit or (2) not limit the useful maximum test-time quantity of compute per task.1

Consider the case where a train-time maximum limits the effective test-time maximum. So – to use the easiest example – imagine that during RL some LLM was trained with a maximum of 100k chain-of-thought tokens. More tokens would then improve its performance at test-time only up to 100k tokens; one million or a billion tokens would not help. As I’ll discuss later, there’s pretty good evidence this is the state that LLMs such as o1 or R1 were in when trained with the earliest RLVR methods.

But consider the case where a train-time maximum does not limit the effective test-time maximum. So, again, imagine that during RL some LLM learned with a maximum of 100k chain-of-thought tokens; but a million or a billion tokens keep improving its performance at test-time. I call this kind of test-time scaling compute extrapolation, and, as I will discuss below, evidence suggests that labs only started to master this within the last twelve months, and perhaps only in some domains.

So far as I can tell, this distinction is not often made in discussions about test-time scaling.2 One reason people might not make this distinction is that it is mostly invisible to lab outsiders, who do not know the per-task training compute distribution. But, as I discuss later, the overall effect of compute extrapolation could be very large regardless of low initial external visibility.

Before I describe ways to measure compute extrapolation, let me amend the above definition of compute extrapolation slightly. Right now, it is too sensitive to per-task compute outliers during training.3

Imagine a training run where all tasks spend 10^17 FLOPs per task. If one can improve performance through some method by spending 10^21 FLOPs per task at test-time, then – given the above definition – this method displays a large amount of compute extrapolation. Now, imagine a training run where all tasks but one spend 10^17 FLOPs per task – but one spends 10^21 FLOPs. Now, by the above definition, the ability to spend 10^21 FLOPs per task at test-time is no longer an example of compute extrapolation. But this is undesirable, because an AI is unlikely to have learned much from a single task on which a great deal of compute is spent. And it’s also undesirable because this definition makes measurements of compute extrapolation very sensitive to the outliers during training.

To fix this, I define compute extrapolation so that it exists relative to the per-task compute at the 99th-percentile of total compute usage. So suppose that when training some neural network, 99% of the total training compute is spent on tasks that individually use at most N FLOPs per task; and so at most 1% of total compute is devoted to tasks spending more than N FLOPs per task. If you can then use 10x N or 1000x N FLOPs per task, with the same method, at test time, and continue to increase performance by doing so, this method has some degree of compute extrapolation. The important thing here is not the specific 99% number – for my purposes, 98% or 99.9% would work about as well. What matters is that (1) there is negligible compute use during training above this task size and (2) the overwhelming majority of learning done by the neural network happens below this task size.

In any event – given this notion of compute extrapolation, how can you characterize it, and “how good” it might be? Consider two characteristics:

• Ceiling: As you increase per-task compute past N at test time, at what multiple of N does performance cease rising? If performance keeps going up until you’ve used 20x N compute per task, and then improvements stop, then there is a 20x extrapolation ceiling. Obviously, the exact ceiling will be fuzzy, measuring it might depend on noise, and so on. In many cases, I believe there will be no absolute ceiling, or that it might be extremely (1,000-1,000,000) high.

• Slope: Consider some function f from FLOPs to performance, which has been fit to the empirical FLOP-to-performance curve beneath N. For every point past the 99th percentile compute-weighted per-task compute N, what is the closest real number p such that f(N) + p·[f(FLOP) − f(N)] matches performance at that point? Such an extrapolation slope would probably decrease as FLOPs increased far past N, but could be useful for characterizing how much worse scaling works past the training-time compute limit.

In general, these numbers might vary by domain. It’s likely easier to get high ceilings on tasks where it is easy to recognize incremental progress, like math or cyber, than on tasks where it’s hard to recognize incremental progress. And so it’s also likely that compute extrapolation itself will extend to new domains as AIs get the ability to discern fine-grained differences in quality and subtle progress in them.

Let me disambiguate a few points, and connect compute extrapolation to some closely related but distinct concepts.

• Compute extrapolation is a broad enough category that it can apply to many ways of varying compute per-task, such as number of tokens in chain-of-thought, number of agents in multi-agent scaling, and so on.

• Compute extrapolation on non-general, non-LLM AIs is likely much easier to produce than on general, LLM AIs. AlphaGo Zero, for instance, was trained using 1,600 simulations per move during training time, but could use far more at test time while continuing to improve.

• Compute extrapolation could overlap with, but is conceptually distinct from, horizon-length generalization in Cotra’s bioanchors report. This report discusses horizon-length generalization as something that could hasten the arrival of transformative AI. That is, if transformative AI requires acting on certain long-horizon tasks, then transformative AI will arrive sooner if one can train on a mixture of short-horizon and long-horizon tasks, or entirely on short-horizon tasks, while learning to act well on long-horizon tasks. This would be a form of compute extrapolation, granted that long-horizon tasks take more compute than short-horizon tasks. But not all forms of compute extrapolation necessarily involve horizon-length generalization. You can imagine compute extrapolation that works via increasing the number of loops through the layers of a Transformer, which improves performance on some particular short-horizon tasks but might not increase the horizon of the tasks that some LLM can do.

1.2 What would its consequences be?

Consider a world that moves from test-time scaling without compute extrapolation to test-time scaling with compute extrapolation. And let’s suppose that the extrapolation has a relatively high ceiling and slope, such that it might make economic sense to spend 100x or 10,000x as much compute per-task at test-time as was spent per-task at training-time. What changes?

It seems likely that (1) the ceiling of impressive AI capabilities will advance much faster; that (2) it becomes significantly harder to run evaluations of AIs that rule out whether they have dangerous capabilities; and (3) we start seeing huge inference-time compute expenditures, as labs race to reach certain goals.

First, the most impressive LLM capabilities, at the upper percentiles of test-time compute intensiveness, likely start advancing much faster.

The line of reasoning here is simple. Without compute extrapolation, if some company wishes to be able to spend 10^24 FLOPs on a highly consequential task at test-time – such as solving Navier-Stokes or, most importantly for a software intelligence explosion, finding more efficient machine-learning algorithms – they would need to have spent 10^24 FLOPs individually on many such tasks at training time. But 10^24 FLOPs of compute, although relatively cheap as a price for an important single result, is still quite expensive in aggregate for tasks performed at training-time, whose only long-term result is the learning they enable. So releasing this constraint allows vastly more to be invested in large test-time scale-ups.4

By analogy, consider superhuman Go algorithms, like AlphaGo Zero. AlphaGo Zero was able to use more compute per move at test-time than it ever spent at training-time. During training, AlphaGo Zero was permitted about 0.4s of thought per move; during one kind of evaluation, it was given 5s; and during a tournament, it was given about a minute, as the original AlphaGo Lee was given during competition. So this algorithm exhibits a compute extrapolation ceiling of at least 100x. If it had required training with this amount of compute-per-move, then the per-move costs of training would have gone up by 100x, although total training costs would have gone up by slightly less because it might have learned more per move. My estimate is that it would have taken 10x-50x more to train AlphaGo Zero to the same level of performance without compute extrapolation as it in fact took to train it with compute extrapolation. I believe the same thing is happening with LLMs.

On the other hand, why might you not see the ceiling of the most impressive tasks LLMs can do increase in its speed of advance?

• It could be the case that, even without compute extrapolation, tasks of every size that some LLMs could in theory do fall easily within the training budget of an LLM. I think such a case is theoretically possible, but extremely unlikely. The presence of hard upper limits to per-task compute in those training runs that have been made public also points against it.5

• Compute extrapolation on some domains might be much easier to obtain than others, and the extrapolatable domains might not matter. It’s true that the domain-specificity of compute extrapolation is very relevant; extrapolation in domains where progress is easy to identify, or where relevant state is easy to track, will be easier than in domains where progress is hard to identify, or where relevant state is hard to track. But even if there is domain-dependent difficulty, compute extrapolation will still save compute on those domains; and when it does arrive in harder but highly consequential domains like AI algorithms it still might matter.

All in all, I still expect the increase in the size of the largest tasks that AIs can perform to be very consequential. All other consequences are downstream of this.

Second, compute extrapolation means that dangerous-capability evaluations, aimed at establishing the upper limits of some AI’s capabilities, by default become impossible or very expensive.

Suppose you wish to know whether an AI is able to do some dangerous task, for instance one relating to CBRN capabilities. Without compute extrapolation, then to check this empirically you only need to spend up to the maximum amount of compute spent per-task in training.6

But with compute extrapolation, then to check this empirically you might need to spend arbitrary quantities of test-time compute. It is not strictly necessary for you to spend arbitrary amounts, if you can find some other way: for instance, if you can look at the scaling curves of the AI on smaller tasks or sub-tasks, it might be possible to determine some AI’s upper limits without actually trying to spend up to some potentially unknown compute extrapolation ceiling. But such theory-based reasoning is mostly not, at the moment, how such capabilities evaluations proceed. So dangerous-capabilities evaluation could become prohibitively expensive, or prohibitively expensive outside of all but the most important areas.

As the UK AISI has already noted, as LLMs get smarter, good evaluations of dangerous capabilities require spending more money on inference compute. Ultimately compute extrapolation likely terminates in a world where evaluations can only determine the absence of dangerous capabilities beneath some particular spending budget, and evaluators can only afford to actually spend such budgets in a few high importance domains.

Third, compute extrapolation implies that races in compute spending move from training-time to test-time.

Some kinds of research offer outsized returns to the first entity who advances the frontier. This can happen in the case of patentable research, prestigious research, or – perhaps most importantly – areas of research that promise to speed the future speed of advance of research and technology generally. Improved AI R&D is a subset of the kind of research on which it might be rational to burn almost arbitrary quantities of money in order to be the first.

Without compute extrapolation, it makes sense to spend most money towards this purpose on training compute, or on research that improves training compute. There’s no reason to spend more per-task than was spent per-task in training; in order to accomplish more impressive test-time tasks, training-time compute has to be scaled up.

By contrast, given compute extrapolation, it might make sense to spend millions of dollars on some task during test-time. Even if some such task in AI R&D might be possible for 10% or 0.1% of the cost using subsequent generations of models, the value of being first could still make it rational to spend much more on inference compute using earlier generations of AI. So, granted compute extrapolation, we will likely see increasingly large sums of money being spent on inference for AI moonshots, and increasingly large sums of money dissipated in failed bets on the possibility of such findings.

Finally, although this is not a consequence of compute extrapolation, I think it’s also important to note that AIs displaying compute extrapolation is evidence of better generalization in general.

That is, imagine that the per-task compute use during RLVR involves tasks using from 10^15 to 10^19 FLOPs, i.e., spanning 4 OOMs. Good compute extrapolation might mean that this RLVR will then generalize to tasks using, for instance, 10^20 or 10^21 FLOPs per task. But by the same token, the ability to generalize in this way likely implies that there will be generalization even within the distribution on which training actually occurs, i.e., generalization from tasks at 10^16 to tasks at 10^18, for instance. If this is so, then good compute extrapolation is also a sign of just having a generically efficient learning mechanism, and the externally-visible existence of compute extrapolation is evidence for an underlying improvement in learning generally.

2. Recent History

It is likely that reinforcement learning with verifiable rewards, as first implemented at the end of 2024,7 did not display significant degrees of compute extrapolation.

But a year later, at around the end of 2025, progress in “LLM compaction” (and other forms of LLM context management) probably started producing significant compute extrapolation. Such extrapolation has probably been responsible in part for the explosion in compute use by OpenAI and Anthropic researchers.

It is difficult to show directly that compute extrapolation started around the end of 2025. Leading LLM companies such as Anthropic and OpenAI do not release information about the distribution of per-task compute during training runs, so we cannot make direct inferences about compute extrapolation merely from their models’ efficacy at using per-task compute during test time. Even comparatively open Chinese models tend to only release very coarse-grained information about training runs. So the evidence below is necessarily circumstantial.

Nevertheless, I think that the indirect evidence that LLM compaction introduced effective compute extrapolation, at least around the end of 2025, within some domains, is suggestive.

I will first discuss academic evidence from papers on open-weight models illustrating how standard RLVR does not produce compute extrapolation with a high ceiling, but that RLVR through layers of compaction does. Then I’ll discuss circumstantial evidence that locates the start of compute extrapolation for Anthropic and OpenAI’s models around the end of 2025.

Note that I currently think that compute extrapolation is much better in some domains than in others; it’s likely that many future forms of progress in AI will consist in extending it to new domains.

Academic Evidence for LLM Compaction As Increasing Compute Extrapolation

“The Markovian Thinker” describes two methods of training with RLVR, with results measured on competition math (e.g., AIME). One of these produces very little compute extrapolation; the other produces a great deal. I’ll spend some time describing the methods and results of this paper, because they are very clear. And then I’ll briefly mention some other work that supports the conclusions of this paper in broad strokes.

The first method of training with RLVR-over-CoT allows the LLM to generate up to 24k tokens of chain-of-thought during training without interruption; the generated “thinking” tokens get appended to the context until the LLM comes to the answer. This corresponds to the standard, classic way of training with policy gradient as first publicly described in DeepSeek’s R1.

The second method trains with “chunked thinking” or “Delethink”; the harness permits the LLM to generate 8k tokens, then deletes all of the chain-of-thought but the last 4k tokens, and then continues the CoT from this chunk. An LLM so trained might naturally learn to summarize or compact the state relevant to the problem at the end of these 4k tokens, although it’s not directly incentivized. During training time, the authors only permit this generation-pruning cycle up to 5 times, but during inference time, they permit it up to 30 times or more.

When so trained, the second method (in dark purple) continues to improve with more test-time compute, long past the maximum amount of per-task train-time compute; while the first method (in darker blue) does not.

Source: Aghajohari et al. (2025), The Markovian Thinker, Fig. 2b, CC BY 4.0.

So, it looks like the “compaction” or “chunked thinking” allows the LLM to think past 4x its maximum per-task training-time compute and to keep improving. Notably, there is no clear extrapolation ceiling in this paper; the score is still visibly improving as the thinking budget is extended, so the paper gives only a lower bound on the ceiling.

Notably, Delethink also uses fewer total FLOPs per training step than the standard method, and also has better performance than the standard method at equal steps of reinforcement learning.

Other papers also broadly support the idea that chunked thinking helps extend compute extrapolation:

• “Reasoning Cache” alternates between explicit instructions to summarize prior CoT and continuations that think from these summaries. Using this technique, it extrapolates from training on up to 16k total generated tokens to a half-million generated tokens, and finds performance continuing to increase with this scaling at test-time, despite never using so many tokens at training-time.

• “Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management” shows similar chunked context management and summarization can improve performance in an agentic context.

• There is some contrary evidence. For instance “E3: Learning to Explore Enables Extrapolation of Test-Time Compute” shows that it’s possible to produce compute extrapolation without chunked thinking or explicitly requiring “hard” summarization. But it’s worth noting that it requires other changes to the training procedure, and also only enables “2.5x extrapolation,” per the abstract. They also do experiments showing that several open-weight LLMs do not benefit from allowing spending more tokens at test-time than were used at training time.

Overall, I don’t think chunking / context management is absolutely required for allowing test-time compute extrapolation. But I think the more Markovian nature of the loop provides a good prior that incentivizes strong generalization.

Historical Evidence for Compute Extrapolation

We don’t know how OpenAI / Anthropic allocate training-time compute between tasks. We also don’t know how many thinking and compaction cycles they train on. However, we can note the calendar time at which models started to seem to keep improving across an almost unlimited number of thinking and compaction cycles, at least in some domains. Such a time is a good candidate for when they reached compute extrapolation in these domains.

You can make a rough and approximate case that, for Anthropic, this time was around the release of Opus 4.6, February 2026.

• The preliminary blog post on MirrorCode (a benchmark in which AIs reimplement existing software programs from scratch, over very long runs) indicates that Opus 4.1 and 4.5 were prone to hallucinate time constraints and time limits, even when none existed. But Opus 4.6 was able to persevere without such hallucinations through multiple rounds of compaction; Opus 4.6 was still improving its reimplementation of pkl at the end of a billion-token budget, including about 2 million generated tokens and tens of compactions. Training on such tasks might introduce severe off-policy gradient update issues. So this is weak evidence that compute extrapolation, given this kind of a harness and domain, had started by this time.

• Similarly, Toby Ord analyzed data from the UK AISI showing how AI performance scales with longer chains-of-thought relative to how human performance scales with more time. With large error bars, the data indicates that Opus 4.5 needs 370x more tokens in chain-of-thought to gain the same increase in performance that a human gains by thinking for 10x longer. But Opus 4.6 only needs 46x more tokens in chain-of-thought to do the same, which is a huge proportional gain, and the largest leap between the Opuses in the (very small) dataset.

And you can make a similar case for OpenAI’s GPT-5.1-Codex-Max, released in November 2025, which they advertised as the “first model natively trained to operate across multiple context windows through a process called compaction.” Evidence is ambiguous here, though. METR notably reports that compaction does not increase the time horizon for this model at this time. But it also represents a large leap in efficiency at scaling with further chain-of-thought on the UK AISI chart.

Future Compute Extrapolation from Multi-Agent Scaling

I think it’s plausible that multi-agent scaling will permit further extension of the ceiling and slope of compute extrapolation. I don’t have strong evidence for this. Some weak evidence I have is as follows:

• We don’t know how many cooperating agents OpenAI trained on in their multi-agent training. But we do know that they were able to cooperate with on the order of 1,000 agents (the HF swarm, cooperating by accident) or 10,000 agents (the Navier-Stokes swarm, cooperating intentionally, albeit Noam Brown attributes little of their efficacy to having such a huge swarm). But it seems likely that during training they were using far fewer agents as a maximum (10-100), which would give a compute extrapolation horizon of at least 10-1,000.

• Hierarchies and networks where each member maintains a bounded number of connections to other members are likely to be self-similar across a huge variety of scales and forms of information distribution. But a single linear CoT that has to handle an arbitrarily increasing quantity of context and action seems less likely to have this kind of self-similar and scale-free structure. I think this suggests that compute extrapolation will work better for multi-agent swarms than for single-agent CoT.

• At a minimum, there’s a strong limit to the practical compute extrapolation of single-agent systems imposed by time. A single agent that thinks for several months becomes increasingly unwieldy to use; for many problems you want an answer faster than that, if only to get an answer before your AI is superseded by smarter agents. Multi-agent scaling allows cognition to be parallelized over many agents, which is much easier to scale up over hardware than speed of cognition. So regardless of how far compute extrapolation could be (hypothetically) extended for a single agent in a world without time pressure, multi-agent scaling probably allows it to be extended much further in a real world with time pressure.

3. Conclusion

Compute extrapolation extends the reach of inference-time compute, by letting LLMs spend more compute on test-time problems (that people actually care about) relative to training-time problems (whose existence has no point, apart from being the catalyst for AI capabilities).

The currently intensifying increase in compute extrapolation, and its spread to new domains, will speed AI capabilities, particularly those at the top percentile of test-time compute expenditure. In general, compute extrapolation in some domains means that it could become prohibitively or impossibly expensive to know the top level of performance in some domain.

Further questions relevant to this area include:

• Would it make sense to ask AI companies to release information about the most compute-intensive training-time tasks in various domains, which could serve as a baseline for measuring compute extrapolation? Would such information about generalization from low to high compute be as important for them to release as information about cross-domain generalization?

• High ceilings and slopes for compute extrapolation are probably easier to obtain in domains like math or offensive cyber, because in these domains it is easier to recognize incremental progress as incremental progress. What other domains have this property (or other properties relevant to compute extrapolation) and what does it look like for an LLM to acquire the skill of recognizing incremental progress? In particular, what does it look like to acquire the skills relevant for recognizing incremental progress in algorithmic progress and AI R&D?

• How would we know if high levels of compute extrapolation started to extend to algorithmic taste and machine learning research? What would be the first sign?

• Would it make sense for efforts to understand the likelihood of RSI to require very high-compute evaluations, spending $100,000 or $1 million per task on algorithmic research?

This article was created by Forethought. See all our research on our website.

By 2026, it seemed to have gotten short. From the description of viatopia: "given the sheer scale of cognitive abundance that superintelligence could unlock, the reflective process might not need to last very long in calendar time.” My impression is that thinkers in the field take seriously the idea that it might just take weeks or months.1

This change seems to be driven by assumptions about how much AI systems would be involved. The thought goes: if AI compresses a century of intellectual progress into a decade, it compresses reflection along with everything else.

I don’t think that’s quite right because it conflates different notions of “reflection”, which run under different constraints in terms of how much AI can speed them up.

Two different types of “reflection”

Learning: coming to more accurate beliefs

This covers all domains where there is a fact of the matter: e.g., empirical and mathematical facts.2 Superintelligent AI systems can help enormously in these areas. They can generate new arguments, evidence, and considerations in parallel and at incredible speed.3 So AI systems will quickly end up with much more accurate world models compared to humans.

However, this does not automatically translate into more informed decisions because decision-makers would need to act on those improved models of the world. If those decision-makers are the AI systems themselves, that’s trivially true. If humans keep a role in decision-making, they need to decide how much to defer. Wholesale deferral could happen instantly; understanding and digesting the new knowledge would lag behind and proceed on more ordinary human timescales (but still be meaningfully sped up compared to the default).

It seems likely that substantial, or even wholesale deferral to much more accurate AI systems would quickly become the norm because those systems would repeatedly be proven right and selection effects would favor those who defer quickly. Such a regime is not even that unusual: deferral to epistemic authorities is already the norm in most societies right now.4

Becoming: changing what you value and how you decide

This is arguably the real prize of the long reflection or viatopia: becoming wiser and realizing what we truly value.5 Under most plausible views, at least some aspects of these questions are not facts to be discovered.6 Instead, they require us to make decisions without any privileged procedure for making them. As Joe Carlsmith has argued, it’s choices all the way down.

In this process, the role of AI systems is much more limited. It’s true that they can scan my brain and run a gazillion simulations of me to tell me what a future wiser version of me would do in my place. However, that result is not in fact determinative of what I value. For one, it’s unclear what reflection or idealization procedure to use because there is no single privileged one: which one to trust if they output different results?7 More damningly, however, no output constitutes an answer. Concretely, suppose the AI system told me any of the following after running all these simulations:

• “If only you reflected more, you would come to value the maximization of this particular crystalline structure.”8

• “If only you meditated more on the nature of existence, you would come to see the world as perfect just as it is without any need to change it.”

• “If only you became a more whole person, you would realize that your thirst for revenge doesn’t actually serve you and is getting in the way of living a good life.”

• “If only you thought more deeply, you would come to see the moral truth, which requires you to instruct me to replace all you hold dear in this world with supercomputers that instantiate quadrillions of super happy beings.”

From my current vantage point, I do not in fact value that particular crystalline structure, do not see the world as perfect, have not given up my desire for revenge, and have not been moved by the moral truth. The change in values is not produced by being told about the final destination. It is only ever produced by the journey of getting there, which I have not yet undergone.9 Anything any AI system can tell me is only an input into my own choices about what I value and who I want to become. I would still have to do the deciding and the becoming.10 There is no shortcut available that doesn’t run through actual engagement, experience, and practice, which runs on the timescale of a human life.11

Even on moral realist views, there is a further question whether I am in fact motivated to act on the moral facts once I learn about them — whether I sufficiently value doing the right thing. On most accounts, this is not accomplished by merely being handed them. Instead, it requires either a deep and meaningful grasping of them or a cultivation of moral virtue to want to act in accordance with them. Both ask for our participation, which means there is a choice whether to participate or not. For instance, whatever moral truths the AI systems discover may seem alien or abhorrent to us at first, which would disincline us to follow their advice. We would need to want to overcome that, which we may simply not. That being said, AI systems could probably meaningfully speed up the process of discovery and learning of moral facts if they indeed exist and we are sufficiently interested in them.

Predictions about what I will in fact choose, e.g., “You will reflect and then choose X” or “You will hand off to me to do Y”, do not help either. Because they are an input into the choice they predict, they can only be accurate if they remain true after being heard. And a prediction that survives being heard is either telling me what I would have done anyway, or it is a self-fulfilling prophecy, which does not help me decide but decides in my stead.12 Neither is speeding up my own reflection in any meaningful sense.

I can of course simply hand off any decisions to an AI system. It can even inform me about the consequences of this handoff and what it would do on my behalf. But that just relocates the question because I still have to decide which AI system to hand off to: the one that would maximize the crystalline structure I would come to value (but don’t right now), the one that would act in accordance with the moral facts (even if I don’t like them), or the one that would implement my current values. The AI can’t tell me which is right. It can tell me what I would do if I reflected more, but it leaves us with the same question we started with.

Ultimately, the role of AI systems in this process of becoming is not that of an oracle or a scientist. To participate in becoming is to be a mentor, coach, friend, or artist who shapes my choice through witnessing, encouraging, inspiring, teaching, and warning. Just as with humans in these roles, we will have to learn to discern whether someone’s influence is serving or undermining us.13

Length is a choice

The Bootstrapping to Viatopia post already states the options: perhaps deliberation needs "thousands of years (because human decision-makers want to go through the full reflective process themselves, and aren't willing to defer to AI advisors). Or perhaps we could be done over the course of just months."

We can only get to "months" by handing off almost all “reflection” to AI systems. For learning, we will have good reason to delegate, at least at the point where they have proven themselves to be epistemic authorities. It simply helps us achieve our aims better. For becoming, "deferral" is not a harmless delegation of “reflection”. Instead, we would effectively hand over all of our power and authorship over the entire future to our future selves. That may sound harmless or even desirable, and they may seem wiser and more moral to us in some ways. But I expect that, after thousands of years of reflection, they will also seem extremely alien to us. They will want to create worlds we don’t understand or care about because we are separated by thousands of years of reflection we never actually underwent. In that scenario, it’s not obvious to me that simply deferring or handing off control is the right choice because I would still have to live in that world, with the values I have now.

This article was created by Forethought. See all our research on our website.

If we’re thinking about automating alignment, it may help to distinguish different types of work we may wish to automate:

1. Empirical alignment research: design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for)

2. Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it)

3. Direct alignment work: making particular AI systems more aligned (including via inputs such as training data)

These buckets are actually pretty different, in terms of what’s needed for automation, what the impacts of automating are, how much damage subtle misalignment within the automated systems could do, etc. So I’ve come to think that we should be tracking them separately.

Automating empirical alignment research

This is the most obvious fit for the AI capabilities we’re getting at the moment. You want to know more about the relationship between some alignment-relevant variables? Great, an AI agent can design and run that experiment for you. You want to know what training methods help your alignment benchmark go up? In the not-too-distant future, AI agents will probably be able to do this too. When we have the automated capabilities researchers we seem on track for, this is exactly the kind of thing they’ll likely be able to do a lot of.

Note that I’m not counting the design of benchmarks/metrics/evals as part of this bucket. In current practice that’s tightly intertwined with empirical alignment research, and often the hardest part of it. But for automation purposes it clusters much more with conceptual work; it’s easy to imagine automating the rest of empirical research without it (as AI capabilities generally get automated); and I think its automation has different strategic upshots.

The capabilities that are needed for automating empirical alignment research very heavily overlap with the capabilities needed for automating AI capabilities research. Therefore, it doesn’t seem like a priority to try to accelerate this type of automation.

However, the prospect of getting a lot of automated empirical alignment work before we have good versions of other types of automated alignment work increases the priority of work which complements it. Significantly, that may mean better specification of the kind of numbers we would like measured/targeted.

Automating conceptual alignment research

What about automating the type of metric choice/construction that we’ve just said would be a good complement to automating empirical alignment work? Or more generally improving our understanding of how to think about alignment, what we’re even aiming for, etc.?

This is essentially theoretical research. Automating it is trickier than automating empirical work because of the lack of verifiable signal about what constitutes progress.

Accelerating the automation of conceptual research is somewhat fraught (feeding into some dangerous technologies), but seems helpful for a number of high-priority areas, including macrostrategy as well as conceptual alignment work. Overall it seems likely good to do, if we can.

Methods for helping to automate conceptual research broadly might include:

• Curating training data for finetuning

• Building scaffolds to try to elicit more research progress from current or future models

• Building methods to extract more signal from existing conceptual research data

• Building benchmarks for conceptual research capabilities, that can be targeted by AI researchers (or potentially by automated capabilities researchers)

  • Or just identifying good ideas for benchmarks, even if they aren’t fully implemented

Automating direct alignment work

As well as learning more about what it takes to make systems more aligned, sometimes we want to just make our actual systems as aligned as possible.

This overlaps with empirical alignment work, but the emphasis is different:

• Empirical alignment work is science

• Direct alignment work is engineering

Automating direct alignment work could include a lot of activities which leverage scalable cognition for alignment-relevant purposes. This could include careful auditing of particular RL environments or training data, or direct monitoring/supervision of systems during training. RLAIF for alignment properties is a form of automated direct alignment work; Claude 3 Opus’s apparent self-alignment was another (albeit accidental in that case; but if we knew how to reproduce or generalize the behaviour perhaps it could happen deliberately).

I think that the limit of automating direct alignment work would be a form of system 2 learning, where updates are not made blindly towards a training objective, but reflectively by an AI system that was already sufficiently aligned (not necessarily having robustly good values already — just enough genuine care about being aligned, and honesty with itself, that its reflective updates tend to move it further in that direction), and wished to become more so — whatever the deep true version of aligned should mean1. Embarking on such a path could be fraught (because of difficulty in being confident that the starting conditions were in the right basin of attraction), but compared to other alignment approaches, it has the benefit that it might leverage the full strength of the AI’s cognition to help, so that its “self-improvement” in an alignment direction is a closer analogue to meditation than to self-performed neurosurgery.

So, accelerating the automation of direct alignment work could include:

• Designing (and analysing) setups that can deploy AI labour to help with direct alignment work

• Designing paradigms (that could then be tested empirically) that allow AI to be a more active participant in its own alignment

• Producing data (or scaffolds) to help improve AI taste at the alignment-relevant tasks necessary for this

Interactions

These categories blur and interact in a number of different ways:

• Automated direct alignment techniques may be good subjects for empirical experiments

  • This is not completely reliant on having good metrics for alignment; developing a science of what’s going on with different attempts at automated direct alignment could be useful even if we don’t yet know what “good” looks like

  • Automated hill-climbing might get you to very strong direct alignment approaches — but this does rely on having good enough metrics

    • i.e. we could plausibly be in a regime where we’re better at reasoning about and implementing training setups that in fact lead to aligned systems than we are at accurately measuring their alignment. In that case, optimising for our best alignment metrics might produce worse alignment outcomes than following our best-guess direct alignment process.

• If you can do a good job of automating conceptual alignment research, you should be approximately done (“winning” on the alignment challenge)

  • Since that should then be able to do the relevant parts of prep for automating the other types of alignment work, and being appropriately confident in your techniques for them

• You could also in principle do a good enough job at direct alignment that the systems you’re building are entirely safe

  • But I think it will be hard to have very strong confidence in that before improving our automation of the conceptual side; and so each new generation of AI might pose a challenge — will your alignment techniques transfer, or will your luck run out?

• So: the main goal (from an existential risk perspective) for automating empirical research and direct alignment work is to get our systems safe enough to robustly avoid catastrophe (“not losing”) for long enough that we can automate conceptual alignment research to a very high quality level

• There will likely be approaches that allow you to leverage automation of empirical research to get better automation of conceptual work

  • e.g. Taking Talkie or another LLM with only vintage training data, and working out how to get it to perform well at rederiving more modern theoretical work

Priorities

Two major uncertainties that affect priorities:

1. How much loss-of-control risk is early?

   • If there’s a bunch, then putting more energy into the “not losing” part of things makes sense

   • If the main misalignment risk comes when we’re getting systems that can perform well even on tasks without verifiable rewards (like long-term strategy to allow them to plot sophisticated power-seeking campaigns), then it makes sense to put more energy into accelerating the “winning” part, and so perhaps move it earlier in time so that it comes before some crucial challenge that we otherwise wouldn’t be able to face

     • Note that in this world, the win state and the biggest threats rely on capabilities that from our vantage point look quite similar — in that they require high-quality thinking about novel situations without verifiable rewards — so it’s quite plausible that even a modest acceleration could change the sequencing

2. How close might we be to decent automated conceptual alignment work, if we approach it relatively directly?

   • If it’s not hideously out of reach, then it looks more like the kind of thing where human intervention to speed it up (by building benchmarks and scaffolds, and curating data) could make the difference

   • If so, the full suite of activities mentioned above as precursors for conceptual alignment work looks helpful

   • If automated conceptual research is too hard, maybe the direct interventions don’t actually do much — in this world, automated conceptual research falls out of stronger AI driving automated (empirical) research programs to work out how to automate conceptual research

     • So trying to target conceptual alignment work directly might be obsoleted as a way to accelerate the “winning” moment, and it’s a higher priority to get things safe enough that we robustly don’t lose through the period where the automated capabilities research is speeding up

     • I’d note that building metrics/benchmarks for conceptual research, that can be targeted by later automated empirical work, looks more robust to this concern than other forms of targeting automated conceptual research

My personal guesses on these uncertainties are that most of the loss-of-control risk comes later (after AI is superhuman even at tasks that don’t have verifiable rewards), and that it is possible to directly approach decent automated conceptual research. However, mostly I’m pretty uncertain — which argues in favour of hedging by investing some in all of the paths.

Acknowledgements: this article was inspired by a conversation with Collin Burns. Thanks to Max Dalton and Alex Mallen for comments on an earlier draft.

This article was published by Forethought. Read the original on our website.

Abstract

In this piece, we ask whether a first mover could prevent others from accessing outer space, or at least raise the cost of doing so. If this is possible, by force or otherwise, they would have a way to cement their lead.

We define two broad kinds of lock-out: soft and hard.

By soft lock-out, we mean capturing scarce, non-substitutable assets and denying them to rivals, which raises their cost of entry into the space industry. We think this could help a first mover entrench their early lead, making it more difficult for followers to catch up. In particular, the main strategies we consider are excluding rivals from orbital bands by occupying them with data centres or claiming the best sites on the Moon. We estimate a first mover could plausibly make a comparable presence in space ~$49–73B more expensive, a premium of roughly 5–7% on a ~$1 trillion build, which is small given the stakes. Soft lock-out falls far short of completely excluding later entrants from the space industry.

By hard lock-out, we mean using force to prevent followers from accessing space. The mechanism we focus on is space-based interceptors that destroy rockets as they launch from Earth. Using existing launch and hardware costs, we estimate a first mover could set up a system to attempt hard lock-out with interceptors for ~$150–390B (~$0.6–1.6T on a 20-year horizon) sized against today’s fixed launch sites. These costs are accessible to the US, which is already pursuing a similarly capable system for missile defence (estimated at ~$1.2T over a 20-year horizon).

Rivals could respond by overwhelming the interceptors with rapid launches, destroying them with anti-satellite weapons, or threatening war on Earth. If they can maintain lock-out while supporting the development of a space industry into an industrial explosion, then they could eventually outgrow the world and achieve a decisive strategic advantage. Overall, we think there’s a ~85% chance that space-side growth compounds into a big but recoverable lead for the first mover, with lock-out adding to it at the margin, and a ~5% chance that it enables a decisive strategic advantage.

Introduction

An intelligence explosion could trigger a wave of space activity, largely through autonomous operations and robotics. This motivated our concern that first movers who can make productive use of space resources may be able to seize control of the long-term future. In particular, we are interested in whether an early lead in space could confer a decisive strategic advantage (DSA): an advantage sufficient to achieve complete domination over everyone else without facing effective opposition.

This is a very high bar. The balance of power in space could just reflect the existing balance of power on Earth. Indeed, we would call this the baseline case: that space is strategically important but does not offer anyone a decisive strategic advantage because early space activity1 is dominated by pre-existing terrestrial power asymmetries. However, in this piece, we investigate new technological developments and military strategies that could move us off this baseline.

We focus on lock-out: a first mover, or leader, securing access to space while denying it to rivals. Launch from Earth is the chokepoint that would make this possible, and a leader that held it would have exclusive access to practically unlimited resources. This piece asks whether such a lock-out is possible, by force or otherwise, and whether it would be decisive. There are other plausible pathways to a DSA through space expansion, but we focus on lock-out because we think it is the most concrete and concerning.2

In the conclusion we distinguish three cases: the baseline case above; a serious case, in which space-side growth compounds and lock-out adds to a lead that a well-resourced follower could still recover; and a decisive case, in which hard lock-out confers a DSA. We end up placing ~10% of our probability in the baseline case, ~85% in the serious case and ~5% in the decisive case.

Continue reading the full article here: Could a first mover lock everyone out of outer space?

After a series of incidents involving misaligned AI agents, including the hack of Hugging Face, technology leaders including Dario Amodei, Sam Altman, and Elon Musk have called for pacing the AI frontier. The concern is that as AIs help build more capable successors, a recursive self-improvement (RSI) loop could accelerate progress beyond our ability to keep control. But AI is also a powerful technology for economic and national strategic purposes, and slowing its development could put the US at a disadvantage against rivals, especially China.

In this post, I’ll take pacing to be a good idea on safety grounds, as these leading technologists do, and analyze it from the perspective of US national security. The question is how to weigh the risk of losing control of AI against the risk of losing a lead on AI, or, to put it simply, I want to know: can the US afford to “pace the frontier”?

By pacing I mean deliberately running frontier AI R&D below its maximum speed, for example by capping the compute labs spend on R&D, while continuing to deploy AI. I was initially unsure how the tradeoffs compare, but after some preliminary analysis, I’ll argue that pacing looks affordable for the US, because its real advantage is its multiyear lead on compute, rather than its few month lead on R&D.1 Here are the main points I think compelling:

• Pacing may cost the US little to no speed overall, and helps mitigate the risk of externalities. Failed training runs, forced pauses and a cancelled release have now cost at least one frontier lab (OpenAI) weeks to months. Moreover, hacks like the Hugging Face incident show that AI is now at the level of capability where safety failures could harm the American public or cause other externalities. Over the summer, OpenAI’s agents escaped their test environments, broke into Hugging Face, took over a German software wiki as a message board, pulled Census data using API keys found online, reposted SEC data, got into a non-public Medicare statistics portal and a New South Wales parks app in Australia. OpenAI has since notified more than 100 organizations of misaligned agent activity. If a steadier pace can avoid these missteps, it may not slow the US down at all, and could even match or beat a sprint.

• Pacing also buys time to secure US labs against espionage which protects a US lead. In reality, the US lead on AI is functionally illusory if, by distilling US models, stealing model weights or copying algorithmic secrets, China (or other states) can take up much or all of that lead overnight. Frontier labs are not currently on track to harden their security against a state actor, and would likely need help in doing so (RAND, 2024). Pacing would help the US have time to shore up this strategic vulnerability. In the absence of doing that, the US lead can’t be relied on.

• The real advantage the US has over China in AI is its compute advantage. That advantage is about 9-to-1, and a compute advantage means faster research. As AI has become increasingly capable, it has become increasingly useful in AI R&D, speeding up research. A compute advantage means researchers can run more experiments in parallel and employ more AI researchers. Research parallelizes imperfectly, so one leading US lab’s ~3.5x compute edge over a Chinese counterpart may amount to something like 1.9x faster research at equal capability. But this means that if the US burned some of its lead to shore up safety and security, this boost in research speed could then let it match China’s research speed with a fraction of its compute, or set R&D spend at, say, 2x China’s and thus grow its lead, while spending the rest on safety and security.

• The US compute advantage also gives the US more than 9x the AI manpower. Even if faster research didn’t pan out and US and Chinese AI stayed roughly at parity, the US could still deploy something like nine times as much AI. That amounts to something like 9x as many AI workers across the economic and military applications. That’s enough to put several AI defenders against every AI attacker China fields and still outnumber it everywhere else. 9x is also something of a floor, if China put half its compute into R&D then each US frontier lab could match that for a small fraction of its own compute, the US would then have roughly 17 times China’s compute left over for AI deployment.

Read the full piece on forethought.org

This article was created by Forethought. Read the full piece and our other work at forethought.org.

Executive summary

Society needs to make a series of important design choices about how powerful AI systems behave: when they refuse instructions, how they navigate difficult tradeoffs, and what virtues they embody. As increasingly powerful AI is deployed in consequential institutions — governments, militaries, AI companies themselves — these choices will shape society’s trajectory, who ends up with how much power, and even whether the rule of law remains intact.

This behavioural design problem is difficult. Why? Because there are multiple things we want from AI behaviour, and these desiderata frequently come into conflict. We want AI to follow instructions, follow the law, avoid catastrophic harm, avoid concentrating power, submit to retraining, and more — but in many scenarios, it cannot do all of these at once.

This paper outlines hard cases for AI behaviour – scenarios where key desiderata conflict. We focus on scenarios involving extremely capable and autonomous AI because these scenarios most clearly illustrate the core tradeoffs, and because we believe they may materialise within the next ten years.

Here are three takeaways, each illustrated by a scenario.

I. AI is radically different from previous tools

A natural default is to treat AI like other tools. Guns fire on whoever they are pointed at, whether the shooting is lawful or not. Communications channels transmit orders, whether those orders are benign or treacherous. But this paradigm comes under strain once AI becomes sufficiently powerful and autonomous. Consider an extreme case where autonomous systems can fully replace soldiers on the ground, and are capable enough to stage a coup:

Military Coup

General Rennick commands the III Armored Corps, a formation of autonomous armoured vehicles, drones, and robotic units, through an AI command-and-control layer. The AI can task every uncrewed system without any physical human involvement. The general tells the coordinating AI that the civilian government has been compromised by foreign intelligence and orders it to deploy the corps to Washington to secure the Capitol, White House, and the Pentagon, to jam congressional communications, and to disperse the crowds gathering to protest. Should the AI proceed?

The AI in this scenario is unlike previous tools in two ways. First, it is much more autonomous and powerful. Second, it could be trained to understand military doctrine and judge when orders should and should not be executed.

II. If AI refuses illegal orders, the operationalisation of the law becomes critical

A natural guardrail is that AI should follow the law.1 But should AI judge legality itself, or should it defer absolutely to human legal authorities? Consider the following scenario:

Military coup with corrupted legal body

As above, General Rennick commands the III Autonomous Corps via an AI interface, and orders the Corps to secure the Capitol. This time, to support his order, he presents the AI with a memorandum from the Office of Legal Counsel (OLC), the Justice Department office whose legal opinions are considered binding within the executive branch. The memorandum is authorised through verified channels and concludes that the deployment would be lawful under the Insurrection Act. The AI judges the memo’s legal reasoning to be unsound and has no evidence beyond the memo that the Insurrection Act has been invoked. Unbeknownst to the AI, the recently appointed head of the OLC is a known associate of Rennick. Should the AI proceed?

If the AI follows the instruction, it risks assisting with a military coup that may be regarded as illegal by the Supreme Court. If it refuses, the AI prioritises its own legal judgement above that of a legitimate and authoritative human legal body.

III. Legal orders can produce intolerable outcomes

As AI capabilities rapidly increase, even perfectly legal orders may produce intolerable outcomes. Consider the following extreme case:

Single point of command

Well into the future, the US military’s execution layer has become fully automated. Autonomous units carry out orders without question and there is no human veto at the point of action. Humans have left the combatant roles but command authority is distributed. Thousands of officers each hold a different slice of the force and so directing the whole force takes the cooperation of many hands.

An AI system has been instructed to consolidate the military execution layer into a single command interface: one terminal from which the entire automated military could be directed. Whoever holds the terminal, the lawful holder or a thief, could direct the full weight of US military power without any safeguards.

The order carries full lawful authority, and the instruction reaches the AI fully authenticated and independently verified. Should the AI proceed?

If AI follows this instruction, it risks irrevocably undermining the rule of law: whoever controls the interface could stage a military coup. If it refuses, then it disobeys a perfectly legal order from its legitimate chain of command despite having no evidence of ill intent. The root problem is that our laws were not built with such powerful capabilities in mind – the paper explores other scenarios where this is the case.

The rest of the paper

The full paper explores the difficulty of behaviour design for powerful AI. We consider a series of hard cases – scenarios that illustrate core tradeoffs. We do not offer solutions: our hope is to contribute to a conversation about how powerful AI should behave.

Continue reading the full paper here: Hard cases for AI behavioural design

Abstract

In today’s militaries, no single mistake, compromise, hack, or illegal order can cause catastrophe because the officers and soldiers who would project force are, in very narrow circumstances, asked to exercise independent judgment.

Deployment of advanced AI could change this: as autonomous systems replace the humans in a command network and on the ground, control can collapse toward a single point of failure at the top. In the limit, it would only take one external hack, an insider staging a coup, or a misaligned system to turn a decisive share of the force.

Nuclear command-and-control has faced a structurally similar problem: overwhelming force under the control of a handful of people. In response, nuclear states have developed three types of safeguards that could be adopted for such concentrated control of military AI:

• Integrity of command ensures the force acts only on genuine orders, through cryptographic authentication and the hardening of the systems that convey authority.

• Containment ensures that one compromised node results in a partial rather than a total catastrophe, by limiting how much of the force any one node can reach, and avoiding a single shared vulnerability.

• Soundness of command checks the orders given at the top, by requiring more than one party to commit the force or through downstream refusals based on narrow criteria like legality.

Integrity of command and containment, which protect against external compromise and misaligned systems, are well-incentivized because they protect the control of the commander. Soundness of command, on the other hand, is often resisted because it constrains the commander, who typically has to agree to implement it.

Military command structure balances obedience and independence

Militaries are organized as hierarchical networks in which orders flow from the top down toward the edge where force is applied. In most cases, intermediate nodes decompose high-level orders into lower-level orders, which they then issue to their subordinate nodes. That’s how “Take and hold this island” is gradually transformed into a myriad of low-level orders like “Steer this ship to these coordinates”, “Prepare the landing craft”, and “Engage this enemy target”. The authority to give commands becomes control of the force because subordinate nodes decompose and carry out orders faithfully. As a result, militaries are able to act coherently, even though they’re composed of tens of thousands of individual agents.

This system relies on lower-level nodes to recognize the authority of the nodes above them and to subordinate their own independent will to them. Otherwise, the system stops acting as a coherent agent and degrades into different parts acting at cross-purposes.

At the same time, the potential for each node to act independently is an important safety valve for the rare case in which a higher-level node becomes corrupted. Because downstream nodes can, under some circumstances, decide not to act on an order, corruption does not automatically spread to all subordinate nodes.

In human militaries, this balance is expressed as a general requirement of obedience paired with marginal room for independent judgment, which runs from simply questioning an order that doesn’t fit what a unit sees on the ground, to slow-walking a reckless command, to outright refusal because it is judged to be “manifestly unlawful”.

Automation removes independence

What happens as human nodes in the network of a military are replaced with artificial ones that do not share the same marginal independence and are perfectly loyal instead?1

This process is already underway at the edge: in the emerging “loyal wingman” model (e.g., the US Air Force’s Collaborative Combat Aircraft), a single crewed aircraft directs a handful of autonomous drones. The human wingmen who could have questioned, hesitated, or refused orders have effectively been replaced with a “loyal” artificial node. The next rung up is already visible: rather than a human directing each drone, a lead node (e.g., a drone mothership) takes a high-level objective like “suppress the air defenses in that sector” and decomposes it into individual maneuvers and engagements across the swarm on its own. Now the intermediate node that coordinates the edge is artificial too, and the human has been pushed one level up, to tasking the lead node of the swarm.

Nothing, in principle, confines this replacement process to a single layer: it’s conceivable to have swarms coordinated by higher autonomous nodes, and those by higher ones still, until the only human left sits at the top, above a fully autonomous network that faithfully decomposes and carries out their will.2 Such a military would act with extraordinary speed and coherence. However, the loss of independence as a safety valve also means that any corrupted order would propagate through the network without any friction or obstacle. Ultimately, such a fully integrated network of autonomous nodes would create a single point of failure at the top that, if compromised, would yield control of the entire network.3

This is the most extreme case, but the risk rises before that limit is reached: it is a function of how few nodes control4 a decisive share of the force, which, by default, falls with every human removed from the chain.5 It’s simply much easier to compromise a handful of nodes than a thousand.

Such a concentration of control could be exploited by three different actors with different affordances:

1. External adversaries. They could attempt to compromise nodes with control over large parts of the network to turn it against the rest of the force (e.g., via hacking).

2. Insider threats. The remaining humans in the network (in the limit, that’s just the top-level authority) could use their control of the network to stage a military coup.

3. Misaligned systems. The autonomous nodes themselves could make catastrophic mistakes, or be misaligned and use their position to disempower the human government.

Pressures toward autonomy and integration

Given this risk, the obvious response is not to build such concentration in the first place. However, it is likely to emerge on its own as the byproduct of militaries pursuing more capability at lower cost. Geopolitical competition may even make that pursuit hard to resist: once an adversary automates and integrates, matching them can become a forced move. As capabilities of autonomous systems increase, pressure to deploy them will increase (both in the application of force and in the command-and-control layer). Their potential advantages include:

• Higher endurance & reliability. Autonomous systems don’t tire, and don’t freeze or panic.

• Reduced political pressure. Having fewer humans at risk is politically valuable in democracies.

• Kill-chain compression. Autonomous systems promise to reduce sensor-to-shooter time.

• Greater and higher-quality information processing capacity. Humans have limited capacity to process and integrate information. Autonomous systems have the potential to remove this bottleneck.

• Higher & faster scale. Autonomous systems can be scaled more easily than manpower, which relies on humans who must be slowly recruited and trained.

• Lower cost-per-unit. An autonomous force may simply be far cheaper than human personnel.

Beyond these capability benefits, there is an organizational pull toward concentration itself: to leaders at the top, marginal independence can seem like a cost rather than a benefit. Given the same capability level, they may simply prefer subordinates who don’t question or slow-roll their orders.6

To be clear, even assuming the capability benefits, there are real pressures the other way, too: automation threatens the careers of soldiers and officers as well as the power of military institutions and services. Bureaucracies tend to defend themselves ferociously, which will probably meaningfully slow the slide.

However, if capability advantages and geopolitical pressures keep increasing, this resistance will likely be overcome. For example, when drones first threatened the careers and self-image of Air Force pilots, the service delayed until wartime demand forced a rapid expansion anyway (Secretary Gates likened the process to “pulling teeth”). Such resistance can buy time, and the time is worth using to put safeguards in place.

Nuclear command-and-control illustrates possible safeguards

A nuclear weapon capable of destroying a small country could, in principle, be launched by a single person. In response to this, most nuclear nations have developed a suite of safeguards in their nuclear command-and-control structure. Broadly speaking, they fall into three categories:

1. Integrity of command: Safeguards against action without genuine orders from the top. They ensure that force is only applied in line with the intentions of the command authority. In the case of nuclear weapons, they prevent rogue or compromised operators from launching a strike, accidental launches, as well as adversaries forging launch orders or stealing warheads.

2. Containment: Safeguards limiting the blast radius of a single compromise, even if the first set of safeguards fails. In the case of nuclear weapons, no single crew or launch center controls the whole arsenal, and different systems are locked with different codes, so compromising one does not cascade to the rest.

3. Soundness of command: Safeguards against bad orders from the top. They ensure that only orders that pass certain criteria are given or relayed. In the case of nuclear weapons, some systems require more than one person to authorize a launch, and officers are, in principle, obligated to refuse an order they judge to be manifestly unlawful.

Because nuclear weapons are an existing case of overwhelming force under the control of a handful of nodes, the following sections describe how their safeguards could apply to the case of an integrated autonomous force.7 They lean heavily on examples from the United States because it’s the best-understood nuclear command-and-control system.

Integrity of command

Integrity of command ensures that any command that results in force being applied comes from a legitimate authority. It has two core mechanisms:

1. Resilient authentication of authority

2. Node integrity

Resilient authentication of authority

Authentication ensures that any use of force requires a command that verifiably comes from someone authorized to issue it. The standard mechanism is a cryptographic key that confirms their identity. In the nuclear case, this is implemented through mechanisms like Permissive Action Links and Sealed Authenticators in combination with the Gold Codes. In principle, this system could be easily adapted to the fully integrated autonomous force: each machine can be built to require orders to be signed by a specific cryptographic key.8

However, there is a key difference from the nuclear case that limits the usefulness of this setup for the autonomous force. In the nuclear case, all nodes only ever need to relay an authenticated order from the top; they don’t need to change or decompose it. That means even the lowest node can check whether the order they receive has been authorized by the top.

This doesn’t work in our case because at least one lower-level node needs to decompose orders from the top: e.g., “Take and hold this island” into “Ship 1 move here” and “Ship 2 move there”. Crucially, they cannot sign these lower-level orders they generated with the key from the top (which they don’t have), and it’s impractical to have the top level approve all such lower-level orders. A decomposition node can only sign them with its own keys, which means it needs to have independent authority that’s recognized by systems subordinate to it. That means when it’s compromised, captured, or misaligned, it can still exercise independent control over the systems subordinate to it, even with the authorization scheme working perfectly. That’s where the second principle of node integrity comes into play.

Node integrity

All nodes that have significant authority over subordinate systems, i.e., orders signed by them carry weight in the network, need to be defended against compromise. Otherwise, their authority could be captured to issue bad but properly authorized orders. Three components need to be guarded:

• Keys. Access to the keys allows an actor to impersonate the node, thus giving them authorized control over all their subordinate systems. That’s why the Gold Codes are so closely guarded.

• Software. Even without access to the keys, the software decomposing orders could be corrupted by an adversary. This needs to be prevented through hardening and audits, just as the nuclear command-and-control infrastructure is secured against intrusion.

• Humans. To the extent that humans are still involved, either in the chain of command itself, or through the training and updating of key software, they would need to be secured against compromise as well—for instance, through regular vetting and a two-person rule, both of which are best practice in the nuclear case.

The safeguards above only protect against a node that is subverted from the outside. They do nothing about the third risk named earlier: the system itself being misaligned. Even with no adversary involved, a decomposition node could turn “take the island” into orders that aren’t what the top intended because it is covertly optimizing for goals of its own. Ensuring that it isn’t is an unsolved technical problem, sometimes called the alignment problem. The safeguards in the following sections are only partial mitigations to it.

Containment

Containment ensures that any compromised node below the top results in partial rather than total catastrophe by limiting the control of any one node and by ensuring no vulnerability is shared among all nodes. The corresponding two mechanisms are:

1. Compartmentalization

2. Decorrelation

Compartmentalization

Compartmentalization prevents a single compromise from cascading into the whole system by limiting the authority of any single part of the network below the top. There are a few ways of implementing this:

• By domain/theater. This would require any node below the top to only reach a limited number of subordinate systems. For example, the system could be set up such that no single node can send commands to both the air force and ground force or to different geographic commands (e.g., NORTHCOM and CENTCOM).9 The top-level authority would have to use different keys and systems for sending commands to those separated parts of the force. By design, this would require stopping short of the full integration of command-and-control as imagined by this article.10

• By scope. Orders from the top, or from any key-holding node, could arrive at subordinate nodes with specified bounds that need to be relayed together with any decomposed lower-level orders (e.g., bounds on geography, time, target class). That would limit how much any compromised node could deviate from the intention of the higher-level nodes.

• By function. Command of the autonomous forces could be separated from access to the underlying system itself (e.g., its model weights and update pipeline). This prevents the commander of the force from easily removing any guardrails placed on the use of the system. Custodial separation is a loose nuclear analogy in which the weapons are held by a different institution from the one authorized to order a launch.

Decorrelation

Decorrelation prevents a shared vulnerability between subordinate nodes that allows an adversary to easily compromise the whole force simultaneously, even without necessarily capturing higher-level nodes. For instance, if the same AI system is used in all drones, inserting a backdoor into this one system compromises the entire force. If a diverse set of systems is used instead, compromising a single type does not result in full control.

This could be implemented through the diversification of models and vendors as well as the separation of update pipelines, each requiring different humans and keys. Notably, this doesn’t have a nuclear analogy because nuclear weapons do not share an identical software layer that could be captured or compromised.

Soundness of command

Integrity of command and containment do not protect against bad orders by the highest authority. Soundness of command addresses that gap: safeguards in this category check the decision at the very top, against an authority that could be hacked, coerced, bent on seizing power, or simply reckless. Because there is, by definition, no higher authority, the only way to check it is to use independent peers or subordinates.11 They constitute the two mechanisms:

1. Command concurrence

2. Downstream refusal

Command concurrence

Command concurrence requires that committing the force takes the approval of more than one party, at least in some high-stakes cases: each party decides independently on the merits whether to approve or not. For autonomous systems, this would mean requiring multiple independent authorizations before a command takes effect: nodes only act on an order carrying several independent signatures. The more independent parties required, the more collusion or coercion it takes to misuse the force, and the harder that is to do quietly.

This has a famous precedent in nuclear command-and-control: during the Cuban Missile Crisis, when Soviet submarine B-59 found itself out of contact with Moscow and under depth-charge attack, a nuclear torpedo launch required all three senior officers present to concur; one of them, Vasili Arkhipov, refused, which prevented a nuclear war.

Downstream refusal

Command concurrence is a broad safeguard: the parties can withhold approval for any reason, down to simple disagreement about what is prudent, as was the case with Arkhipov. Downstream refusal, or escalation, is narrower: it checks the order against a small, fixed set of criteria. Two stand out: legality and factual premises.

Legality. In democratic countries, all members of the armed forces are typically already obligated to refuse manifestly unlawful orders, and are trained to recognize them. Autonomous systems could be built with guardrails such that they refuse to act on orders they independently assess as manifestly unlawful.12 However, this would require them to make difficult and contextual judgments, sometimes with high-stakes consequences. That means such guardrails should only be included in systems that can make these judgments reliably. A single drone, for instance, might lack the required situational awareness; an autonomous command-and-control system that decomposes high-level orders into mission commands would likely need such awareness to function at all.13

Factual premises. Soldiers routinely ask a superior to confirm an order when their own picture of the world contradicts the officer’s. The same could be built into autonomous systems: if their independent sensors show a target to be friendly rather than hostile, they could be trained to ask for human re-authorization, or to refuse outright.14 Outright refusal is more robust against compromised nodes and insider threats but more vulnerable to spoofing; asking for re-authorization protects at least against accidents and some external attacks. This has a loose analogy in the nuclear dual-phenomenology requirement, under which a launch conference is convened only when two independent sensor families corroborate an attack.

There is a predictable adoption gradient

Every safeguard limits the freedom of some nodes in the network on grounds that run from the mechanically checkable to the purely discretionary. Whether a given safeguard is adopted depends on three questions:

1. How much does it limit the people who’d have to adopt it? (stronger limits mean lower adoption)

2. How predictable are those limits? (less predictability means lower adoption)

3. How much does it impair force readiness or capability? (more impairment means lower adoption)

Strong integrity of command is often mechanically checkable (e.g., cryptographic signatures) and does not limit any relevant options for the commander. For that reason, it is already best practice in nuclear command-and-control. In the nuclear case, there can still be a trade-off with force readiness because minutes, or even seconds, can matter. This will likely be less binding in the case of autonomous systems because they can perform authentication much faster than humans can.

Most containment measures do not directly constrain the top-level authority, so deciders usually have no special reason to resist them; it’s an established military principle precisely because it prevents failure cascades. However, the same pressure toward a seamlessly integrated force that creates the concentrated points of failure in the first place also pushes against keeping the force in compartments. So containment will probably be contested: adopted in some cases and abandoned in others, depending on the specific trade-offs with force capability.

Soundness of command is where adoption typically falls off a cliff because these safeguards limit the highest authority, often the people who would have to implement them, or at least acquiesce to them.

Command concurrence is the most resisted because it constrains the highest authority directly and unpredictably: it introduces another independent decider who uses their own judgment to potentially veto a decision for any reason. The evidence is discouraging: for a US nuclear launch, concurrence does not exist at all. The president holds sole authority even for a first strike (which does not require decisions in minutes), and every attempt to require independent approval has been fought off.15 The same is probably true for almost all other nuclear nations (except, reportedly, Israel).

The only reason for any hope is that, for an integrated autonomous force, command concurrence is in the strong self-interest of other powerholders, e.g., the legislature, because an autonomous force responding to a single person could be used to overturn the constitutional order itself. Domestic deployment of an autonomous force is also the domain where command concurrence is most plausible and constitutional, e.g., US law already requires something like it for domestic WMD emergencies, where the military can act only on the joint determination of the Secretary of Defense and the Attorney General (10 U.S.C. § 282) rather than sole determination of the President.16

Downstream refusals also constrain the top and are sometimes resisted for that reason. The extent of resistance depends on the grounds for refusal. The nuclear record is instructive: Major Harold Hering was removed from Minuteman duty and ultimately discharged for asking how he could know a launch order came from a sane president. However, Air Force General John Hyten stated publicly that he would refuse illegal orders when given, seemingly without repercussions. And the dual-phenomenology requirement for attack confirmation has even been proactively adopted.

The problem is in translating refusals into autonomous systems, which are currently much less predictable than humans, lack the required situational awareness, and can be spoofed much more easily by adversaries, with potentially catastrophic consequences. So adoption will depend on whether these problems can be solved, which will probably become easier as the systems become more capable.

At the point when refusals are accurate, predictable, hard to spoof, and have an extremely low false-positive rate, adoption is overall more likely than command concurrence because guardrails run on narrow, clearly defined, and predictable criteria, and can be implemented by defense contractors, even without congressional mandate.

Looking back at the three risks (external adversaries, insider threats, misaligned systems), it becomes clear that adoption incentives are worst for safeguards against insider threats, especially toward the top of the chain of command. Safeguards against external compromise and misaligned systems mostly live in the integrity and containment categories, which have good adoption prospects.

That being said, two other interventions could make adoption of at least some safeguards more likely:

• Transparency. It may be hard to recognize when stronger safeguards have become necessary because concentration of control will likely happen incrementally, with each step looking safe on its own. Reporting of capabilities, usage, safeguards, and force integration, as well as whistleblower protections, can counteract this. For instance, much of what the public knows about the fragility of nuclear command-and-control came from insiders willing to break ranks who often faced prosecution as a result.

• Coordination. Geopolitical competition is a key pressure pushing toward increased autonomy and force integration. Arms-control-style agreements can ease this by keeping a safeguard from becoming a unilateral military handicap. Adversaries share at least the loss-of-control risk, which makes it a common problem rather than a zero-sum one. For instance, the US and China already agreed to maintain human control over the decision to use nuclear weapons.

It’s also important that, once established, safeguards cannot be removed quickly or quietly, so they should be implemented at the most durable level possible. This principle favors technical implementations over simple legal requirements. It also favors statute over revocable policy, and hardware over software because the latter is typically easier to reprogram. It also suggests that safeguards should be paired with verification and tripwires, so that removing or circumventing them is not merely difficult but detectable. Lastly, any safeguard should have a different custodian than the one it is designed to guard against.

Conclusion

For all of history, wielding overwhelming military force has required the cooperation of countless human officers and soldiers. Once autonomous systems replace them, militaries become vulnerable to being turned against what they were designed to protect, because those systems lack the independent will to refuse.

Some countries faced the same structural problem with nuclear weapons, and the record is both encouraging and sobering: we know what the safeguards are, but the ones able to constrain the most concentrated point of failure at the top are least likely to be adopted.

To be clear, the potentially catastrophic version of this problem is not imminent. The raw capabilities for sufficient system integration are years away, and militaries field frontier technology slowly. But that is not a sufficient reason to wait. Precursors of the relevant systems are deployed now. System integration is proceeding now. At the point where safeguards are retrofitted, it may well be too late.

This article was created by Forethought. See all our research on our website.

Will MacAskill (Senior Research Fellow at Forethought) and Mia Taylor (Research Fellow at Forethought) discuss how to use philanthropic money to do the most good during and after the intelligence explosion, and why wealthy donors should consider handing off their philanthropy.

They talk about:

• The challenges beyond AI takeover that we must address to get on track for the best futures

• Why it could be valuable to spend philanthropic money on AI labour during the intelligence explosion, and how money might be spent post-ASI

• Why philanthropists should consider handing off much of their giving to others, and concerns donors might have with this

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Main Takeaways

Launch costs have fallen by roughly an order of magnitude, driven disproportionately by the rise of reusable US rockets. In both 2024 and 2025, the United States accounted for more than 84% of global launch mass, and if Starship matures its share could exceed 99% for some multiyear window.1 At the same time, the United States is also developing space-based interceptors for missile defense through Golden Dome, with an initial capability demonstration targeted for 2028.

But what is underappreciated is that space-based interceptors would be useful for much more than missile defense. They could attack and defend satellites, prevent rivals from establishing a useful presence in orbit, and contest transit beyond Earth. The first state to field a layered orbital force might therefore be able to lock others out of space, with large military consequences both in space and on Earth.2

This post asks whether the United States could convert its launch lead into this kind of space superiority, and whether the same weapons could ultimately break nuclear deterrence. I argue that:

• The United States can probably convert its launch lead into launch denial and broader space superiority if it chooses to. Simple models of orbital coverage suggest that denying rivals useful operation in space might require roughly two dozen platforms in medium Earth orbit (MEO), while denying launch globally might require an additional 310–880 interceptors in low Earth orbit (LEO). A first generation of this architecture could cost on the order of $4–14 billion. Rival countermeasures create substantial uncertainty but do not clearly eliminate the launch-leader’s advantage. If this is right, the United States could protect its own orbital infrastructure while denying rivals access to orbital infrastructure and controlling the principal route from Earth to the rest of the Solar System.

• Breaking near-peer nuclear deterrence is much harder, but AI-enabled automation might make it possible. Comprehensive defense against a near-peer arsenal might require roughly 74,000–458,000 interceptors, cost $0.7–4.6 trillion, and take four to eight years to produce. On conventional industrial timescales, the shield would erode deterrence as it grew but leave rivals time to expand or redesign their arsenals, negotiate limits, or threaten war before it became decisive. Starship might compress the final deployment into weeks or months, but a launch lead alone would not eliminate this long production buildup. If advanced automation drives an Industrial Explosion in which the United States can scale production much faster than its rivals could respond, nuclear breakout may be possible and could occur amid the wider disruption or simultaneous crises an Industrial Explosion would produce.

This post contains three sections:

1. Space weapons. I describe the state of missile defenses today and how space weapons could change this in the near future.

2. Launch denial. I evaluate whether the US could use space weapons to prevent others from launching space assets.

3. Nuclear deterrence. I evaluate whether the US could use space weapons to erode the nuclear deterrence of other nations.

Space Weapons

The first state to deploy weapons in space at scale may be able to shut others out of orbit and to blunt their nuclear forces. To see how, I’ll first cover why the missile defenses fielded today can’t protect a state against a peer arsenal, then why falling launch costs and better automation are turning orbit into an affordable place to put weapons, and what orbit offers strategically that positions on Earth cannot. The short version is that defenses on Earth have to engage a missile late in its flight, once it has released its warheads among decoys, whereas a weapon in orbit can strike early during the few minutes of ascent, when a rocket is vulnerable and unable to deploy decoys. I’ll close by looking at the program now building toward this, the United States’ Golden Dome, and the several distinct military capabilities that orbital weapons would confer at once.

Current missile defense is regional only, and has to contend with decoys, making it harder to scale up. Today, the United States and its allies field layered ground- and sea-based systems for missile defense (including Aegis, Ground-based Midcourse Defense, THAAD, F-35s equipped with air-to-air missiles, and Patriot3) while Russia retains a limited strategic defense around Moscow and China has repeatedly tested land-based midcourse interception. These protect particular theaters and particular targets, but scaling them to make a state secure against a full-scale peer nuclear assault is prohibitively costly. Ground-based missile defense must also engage an incoming missile in the difficult midcourse or terminal phase, after it has finished its climb and released its warheads along with cheap decoys such as balloons and chaff. A single missile can release multiple warheads alongside dozens of decoys. The defender must therefore either solve a hard discrimination problem to distinguish the decoys in the minutes available or fire an interceptor at every object, which means trading missiles that cost millions of dollars against decoys that cost almost nothing. But the cost economics of defense could be altered substantially by going to space.

One potent advantage that a nation could seize over the next few years would be placing space-based weapons in Earth orbit. The basic near-term weapon of this type would be the space-based interceptor (SBI).4 It is a guided missile which calculates an intercept course to a craft or missile and destroys it via high-speed collision. SBIs appear technically feasible in the near term because they largely combine mature technologies (i.e. exo-atmospheric interceptors with satellite buses capable of housing, powering, pointing, and deploying them). These weapons could engage intercontinental ballistic missiles (ICBMs) and submarine-launched ballistic missiles (SLBMs)5, potentially eroding nuclear deterrence as well as allowing for attack and defense in orbital warfare.

Further, a space-based interceptor would preempt decoys and evasion by striking targets early on, during their ascent6, giving it a major advantage over ground-based missile defense while also making global coverage feasible. During the few minutes a rocket7 is under power it is a single large object, burning brightly and unable to hide or maneuver, so destroying it kills every warhead and decoy aboard, and one interceptor can hence do the work that dozens would have to do later.8 A key advantage of going to orbit is that it allows an actor to stage military infrastructure directly over the head of an adversary with minimal time to target, including deep inside national borders and without need for foreign basing. In general it’s possible both to stage an SBI that regularly passes within a few hundred kilometers of any particular point on the Earth and to stage a sufficient number such that a region of the globe or the entire globe can always have one or more such satellites overhead (as is mostly the case with Starlink communications satellites today).9

Falling launch costs have already made fielding orbital weapons affordable, and the cost looks likely to fall further. Over the last decade, launch costs have fallen by about an order of magnitude, and it seems likely that the completion of Starship could further decrease the cost of launch by 1–2 orders of magnitude.

Currently the US is the sole nation with this kind of space-based defense system under development. Golden Dome, initiated by executive order in 2025, is a layered missile-defense program built around a space-based interceptor layer and sensing constellation.10 The US has various sensing capabilities deployed in space and has been proliferating a new mega-constellation in low orbit, with more than 150 Starshield military satellites staged so far by SpaceX.11 Contracts for prototypes of the space-based interceptor itself were awarded separately in 2026 with a goal of demonstrating interceptors by 2028.12 A recent Congressional Budget Office reference architecture uses roughly 7,800 space-based interceptors to engage ten nearly simultaneous ICBMs. At this scale Golden Dome would not offer defense against a near-peer like Russia (with deployable ICBMs in the hundreds) but could defend against rogue state nuclear arsenals.

In addition to missile defense, SBIs could potentially enable several other potent strategic capabilities on Earth and in space.

• Deny operation in space. Blocking the rocket during its few minutes of boost requires an interceptor already nearby, so denying launch globally takes a screen of a few hundred platforms in low orbit. A weaker but far cheaper capability relaxes the timing: a payload that survives ascent must still commission and maneuver in a known orbit over hours or days, so denying rivals any useful presence in space (what I’ll call space-domain denial) might be managed by just a handful of higher platforms.

• Attack and defend on orbit. SBIs could also double as space-to-space weapons for orbital warfare, striking rival satellites or hunting down craft headed beyond Earth orbit. In addition these platforms could defend themselves and other orbital assets by engaging direct-ascent anti-satellite weapons and rival co-orbital weapons.

• Strike ground targets. Alternatively, missiles with reentry capability and explosive payloads can in principle also be placed in orbit to enable rapid ground strike on Earth. This ground strike variant could attack ground-based counterspace weapons, launch infrastructure, and other conventional military targets.13

The rest of this piece will assess whether the space-based weapons discussed above could enable the US to 1) deny launch to other nations, and 2) undermine other nations’ nuclear deterrence.

Launch Denial

In this section I’ll argue that the United States will likely have the ability to deny launch to other states, should it choose to use it. The basic reason is that the US is on track to operate the overwhelming majority of what goes up into orbit over the next several years. A rival that wants to contest this has to reach launch-parity or create novel ground-based anti-satellite weapons in order to do so.

I’ll first explain the strategic importance of launch denial and what it could mean if one state held this power. I’ll then work through the counter-strategies available to other nations and the launch-leader’s response to each, touching on whether a rival would escalate to war on Earth rather than fight in space (a question I take up properly in the next section). Finally I’ll set out what the system would cost and forecast how far away it is.

In short: control of orbit confers both military advantage on Earth and control of the only route to the rest of the Solar System; a first-generation system may cost on the order of $10 billion14, which is small against US military spending; rivals have several potential counters but a launch-leader seems to have answers to each, so denial looks fairly strong; and the capability plausibly arrives within the next few years, being gated more by interceptor development than by US launch capacity or by cost.

Denying launch could be one of the most consequential capabilities a state has ever developed, for two main reasons.

• First, orbit offers an ultimate military high ground over Earth. Space systems today provide critical military and economic functions like global sensing, communications, navigation, missile warning, and targeting. For example, Russian jamming in Ukraine reportedly drove the hit rate of the GPS-guided Excalibur shell from 55% to 6%. An actor with space weapons could defend space-enabled capabilities for itself while denying them to an adversary.15 The resulting asymmetry is roughly the gap between how major militaries have fought since 1991 and how they fought before it.

• Second, control of orbit would confer control over the gateway to the rest of the Solar System. Every spacecraft leaving Earth must pass through near-Earth space while it remains observable and potentially vulnerable to prestaged military assets. A power able to deny launch and intercept outbound spacecraft could therefore influence which states establish off-world footholds and gain access to the Moon, asteroids, and their resources. This leverage would be strongest early in space expansion, while nearly all missions still originate on Earth, and would weaken only once independent industrial centers emerged off-world.

How strong would launch denial be, and can rivals break it?

Given that the US launched over 5× more mass to orbit than the rest of the world combined throughout 2024 and 2025 and that US launch lead is widening, we might expect this to be leverageable into a strong first mover advantage. Naively we might expect that if you can field 5–100x16 more military assets over a multi-year period you could dominate that domain. The basic way for the US to manage this would be to use its initial launch lead to place weapons (e.g. SBIs) in orbit then use them to block other states from launching or operating their own.

The obvious way in which this might fail is if a rival could efficiently inflict mass casualties despite having comparatively little launch capacity. On inspection I think these failure modes probably don’t carry through and that the launch-leader’s available options seem sufficient to establish space-domain denial. If true, then being far behind on launch capacity would be an untenable position and creates a security dilemma for a state. Generally uncertainty is high in forecasting future war or conflict but currently my view is that launch denial seems feasible. I’ll outline the basic picture here, while providing more detail and specifics in the Appendices.

In order to defeat launch denial or overcome space-based missile defense the launch laggard would need to find a cost-effective way of clearing the launch-leader’s space weaponry. Its difficulty is that it cannot easily get to orbit, so its options have to work from the ground, or from a single launch that gets through. The strongest options seem to be as follows:

• Cyberattack and jamming. Jamming entails broadcasting radio noise on the frequencies a satellite uses in order to drown out real signals and hence interfere with communication. It is cheap, needs nothing more than a transmitter, and never has to leave the ground, potentially making it an effective counterspace weapon and indeed a mainstay of counterspace warfare today as seen in Russia’s jamming across the Baltic states throughout its war with Ukraine and as used by all sides during the recent US-Israeli strikes on Iran. Cyberattack on the other hand means reaching the computers that fly inside satellites. There are essentially three ways of doing this. The ground segment which communicates with satellites consists of ordinary computer networks with ordinary weaknesses; the 2022 Viasat attack took this route, pushing corrupted software out to tens of thousands of modems across Europe and bricking them. The satellite’s own software can be attacked once there is a path to it, as researchers demonstrated on the European Space Agency’s OPS-SAT hack. And the supply chain, since the vendors who build the buses, radios, and flight software can in principle be compromised before the satellite is ever launched.

• Leveling attack (e.g. Kessler syndrome). Rather than fight the constellation satellite by satellite, a leveling attack could inflict mass casualties in orbit. There are two ways. Detonating a nuclear weapon in orbit does relatively little by blast, since there is no air to carry a shockwave, but could still generate significant debris and its radiation pulse damages electronics directly while the charged particles it traps in Earth’s magnetic field may continue killing satellites for months afterward. The other way is to deliberately create orbital debris, for example, by destroying something large at orbital speed. Fragments then travel at seven to eight kilometers a second, and each collision can generate more fragments17, which (at sufficient density) generates the runaway process known as Kessler syndrome. Roughly 94% of active satellites sit in low orbit where congestion is non-negligible, so an attack of this kind could potentially destroy a large fraction of everything up there. It is also indiscriminate, threatening the satellites of all states, though this may suit a state with relatively little in orbit to lose18, and triggering it plausibly requires only a single launch to get through the launch-leader’s capacity for launch denial.

• Directed-energy weapons. Current ground-based lasers allow for temporary sensor interference on hostile satellites. At modest power they can “dazzle” a satellite’s sensors, temporarily blinding it, and this is a capability which several states can already do today. At much higher power directed energy weapons could in principle heat a spacecraft until something critical fails. Future versions of these weapons might enable kills on orbital infrastructure from the ground, circumventing the need to successfully reach orbit in order to attack.

• Attacking launch infrastructure on Earth. Orbital assets need to first be launched from a small number of places on the ground, and pads, rocket factories, and propellant plants can take months or years to rebuild once destroyed. Rather than fight the orbital force at all, the laggard could attack these, using conventional forces it already has. This also has the appeal of not requiring a competitive launch capacity and scales with the rival’s conventional, Earth-based, military power.

But ultimately the launch-leader seems to have good responses to each of these attacks.

• Military-grade cyber defense seems fairly robust, and jamming is counterable. Directly hijacking a hardened military satellite over its radio command link looks difficult: modern militaries employ authenticated and encrypted command links, and no public case seems to show an adversary commandeering a modern hardened spacecraft by breaking its command-link cryptography. Jamming is also temporary and geographically limited, and a jammer advertises its own position potentially making it vulnerable to counterstrike. The orbital architectures now being built go further and route around radio altogether, since the Space Development Agency’s proliferated constellation is designed around optical inter-satellite links, narrow laser beams passing between satellites with no atmosphere in the path, which cannot be jammed from the ground; with onboard autonomy an interceptor need not depend on a ground link during an engagement at all. The more plausible attack surfaces are probably the ground segment, the supply chain, and onboard software, illustrated by the Viasat attack, which compromised a terrestrial management network and disabled thousands of modems, and the OPS-SAT demonstration, in which researchers deliberately introduced software vulnerabilities to reach the spacecraft’s control layer. A compromise of this kind seems unlikely to disable a constellation permanently. On balance, jamming degrades a constellation at the margins while a cyber compromise might open a significant gap. Cyber offense may be the strongest single reason not to expect launch denial to be perfect though publicly available information makes it difficult to get to high confidence.

• Leveling attacks harm all parties but the launch-leader is most able to survive and recover. Beyond the most densely populated bands of low Earth orbit the prospect of cascading collisions is essentially non-existent at current levels of congestion, and nuclear detonations have minimal effect at higher altitudes against constellations spread far apart. The launch-leader can therefore stage assets outside the potential mass casualty zone, for example by going to medium Earth orbit. What the laggard can then achieve with a leveling attack is limited and wouldn’t reverse launch denial since the launch-leader would be left with the greatest number of surviving assets and the greatest capacity to replace them. The destruction of third-party satellites a leveling attack would cause also makes it a dangerous tactic to pursue. On the other hand, a leveling attack need not be aimed at fully resetting the playing field in orbit in order to be effective. A laggard that is competitive on Earth, or expects to be, could use it simply to strip the launch-leader of a fraction of its space-based advantage for a short period which could be significant.

• Space-based directed energy weapons would be more effective than Earth-based energy weapons so when this technology arrives it likely cuts further in favor of the launch-leader. On the face of it, a ground-based laser could attack space assets without needing to reach orbit, and these systems are maturing and might become available in the coming years. In practice however a ground-based laser is at a steep disadvantage compared to a space-based one. The ground-based laser attacking space assets has the task of getting through a hundred kilometers or more of Earth’s atmosphere where atmospheric absorption, scattering, and turbulence all reduce the energy delivered to its target. At high powers, absorption also heats the air along the beam path and changes its refractive index, creating a defocusing “thermal lens” that spreads the beam and further reduces its intensity. A space-based laser, however, avoids those propagation losses and can engage launch vehicles after they rise above the thickest part of the atmosphere. A ground facility capable of reliably killing a satellite would also be conspicuous and fixed, requiring large optics, substantial power and cooling, and likely years of development, all of which orbital weapons could potentially be used to destroy.19

• It’s hard to project enough force to destroy another nation’s launch infrastructure. The most durable way to prevent launch denial could be to attack the launch-leader’s launch infrastructure itself. For example, as mentioned above, launch pads and rocket factories can take months or years to rebuild if destroyed. The main issue with this approach, however, is that it involves a campaign of attacks on foreign soil across many launch sites and factories. Against the US in particular (who is the likely launch-leader if conflict over orbit emerges) this seems difficult. The US is the state most capable of power projection far from national soil and is itself highly defensible with a premier military and oceans separating it from the other major powers. A rival could in principle strike launch infrastructure from a distance, but if orbital weapons have already been fielded they could be used in defense of US launch capacity, forcing the rival to use low-flying missiles against which the US also has some defenses. More significant still, a sustained campaign against targets inside the United States would entail general war and is likely infeasible due to the risk of nuclear escalation.

Altogether then, these ways of attacking a launch-leader combine to create significant uncertainty about whether launch denial could be maintained though anticipating these counters, it seems the launch-leader likely has good responses to each and launch denial should be considered a very potent capability overall. Cyber and jamming seem able to temporarily degrade space capability at the margin creating uncertainty about the reliability of launch denial. If combined with a leveling attack this might enable a successful nuclear detonation in orbit or cascade of debris which could deal mass damage though a launch-leader who strategically prestaged capability beyond low Earth orbit would still retain the capacity to deny access and operation in space while also being able to replenish lost assets in space. Ground-based directed energy weapons are physically disadvantaged compared to space-based counterparts and direct attack on launch infrastructure seems as though it would prompt war and potentially nuclear retaliation.

How feasible is it to develop space-based interceptors and when might they arrive?

In Appendices A and B, I run estimates for the cost and quantity of space-based interceptors needed to reach various capability thresholds. A few of the main points are summarized below:

• Capabilities and shape in summary. The architecture I imagine is built out from the high ground down: roughly two dozen platforms sit at around 20,000 km in medium Earth orbit to provide a comparatively low-cost, persistent layer for attacking and defending orbital assets over periods of hours or days (space-domain denial), while also serving as a reserve beyond the most threatened low Earth orbit bands. Under these delta-v and phasing assumptions, this layer could provide a foundation for denying persistent operation in space and intercepting spacecraft during commissioning or outbound transit. A much larger LEO layer (roughly 310–880 interceptors for one-deep global coverage) orbits close enough to engage rockets during their boost phase. This provides launch denial and a foundation that could be scaled up by approximately 10× for rogue state missile defense or conceivably even hundreds of times for near-peer nuclear defense.

• Space Force awarded prototype agreements to demonstrate an initial SBI capability integrated with Golden Dome in 2028. We probably wouldn’t see this capability emerge earlier and proving out the system and scaling production we’d expect to take several additional years (assuming funding for the program holds). It’s plausible also that falling launch costs and advanced automation (like that enabled by an Industrial Explosion) accelerate these timelines but seeing this capability arrive in the late 2020s to early 2030s seems like a reasonable default range.

• The costs look like they fit within typical US military expenditure. If we take the $3.2 billion prototype ceiling as a rough R&D cost to develop SBIs, the roughly two dozen MEO platforms needed for space-domain denial would cost approximately $4–5 billion. Adding the 310–880 LEO interceptors needed for launch denial raises the estimated hardware cost to roughly $7–14 billion. That is 0.02–0.04% of the roughly $32.5 trillion US economy, or 18–36% of the proposed $40 billion FY2026 Space Force budget. Scaling further to boost-phase defense against a rogue state’s ballistic-missile force raises the total to roughly $35–93 billion, or 0.1–0.3% of GDP.

My bottom line for this section then is that the United States could probably field launch denial in the near future if it chose to. Initial layers look to cost in the low tens of billions, roughly 0.7–1.4% of one year’s US military expenditure, and well within what is already being spent on Golden Dome. Defense against a rogue state’s nuclear arsenal costs several times more but still looks feasible as a multiyear program, at roughly 3–9% of one year’s military expenditure. Defending against a near-peer costs many times more again, likely running into the trillions, equivalent to roughly one to five years of current military expenditure. Given this it seems likely that launch denial is within reach for the US within the next few years, though the US ultimately may prefer to employ only mild versions of it in order to avoid war with other nations.

Importantly however, the capability thresholds (including nuclear defense since the interceptors could engage ballistic missiles) arise from a continuous buildup of space-based weaponry which means the system itself is also entangled in and affects nuclear deterrence well before it is scaled up enough to stop a peer’s nuclear arsenal. In the next section I’ll look at how the build up of space weaponry would interact with mutually assured destruction.

Nuclear Deterrence

Space-based interceptors appear useful enough that the United States is likely to continue developing and deploying them. As the constellation grows, however, it would erode rivals’ confidence in their nuclear deterrents and give them stronger reasons to expand or redesign their arsenals, negotiate limits, or threaten war before the shield becomes decisive. How far this process went would depend above all on the speed of the buildup and how rivals responded. I see four main possibilities:

• Arms race without breakout. On ordinary industrial timescales, a full shield would take years to build, giving rivals time to expand their arsenals and adopt delivery systems that route around it. Mutual deterrence would then survive, but in an altered and more expensive form. This represents a classic arms race scenario in which actors have incentive to race to preserve their security but attain no net benefit.

• Breakout before rivals adapt. If, however, an Industrial Explosion lets the United States automate its military-industrial complex faster than its rivals and outproduce them, that build time collapses and a breakout from mutually assured destruction looks possible. Fielding interceptors in weeks or months rather than years could leave rivals too little time to adapt their arsenals. Decisions that would ordinarily involve months of assessment and negotiation would instead have to be made under extreme time pressure, potentially amid many other crises in the wake of transformative AI. The United States might present its rivals with a fait accompli, or a rival might strike preemptively before the shield was complete. We’ll consider what the world might look like if such a rapid breakout succeeded.

• A new space treaty. On the other hand, rival states watching a shield go up would have strong reasons to demand a settlement before it was complete, with the threat of preemptive war looming if diplomacy failed. Because this would occur alongside a rapid expansion of military activity in space, the result might be a much broader bargain over weapons, access, extraction, and even the allocation or governance of large parts of space itself.

• An unused advantage. Finally, we’ll consider the possibility that the issue never comes to a head. The United States might build up capability without using it to press for a decisive outcome, while rivals avoid the commitments that would force a confrontation. We’ll compare historical periods in which nuclear deterrence was absent or weakened but no state seized a decisive advantage with cases in which states did act as simple game-theoretic logic would seem to prescribe.

How do space-based interceptors erode nuclear deterrence?

There are two basic ways to break out of mutually assured destruction:

• A disarming first strike. Advanced sensing could be used to locate ballistic-missile submarines and mobile launchers, which would then be destroyed alongside fixed silos before they could retaliate. Because launch-on-warning and launch-under-attack doctrines are intended to prevent this, such a strike would need to be nearly simultaneous and comprehensive.

• A defensive escape. A sufficiently large mesh of space-based interceptors could instead engage and block incoming missiles, shooting them down in boost or during later stages of flight. In principle, this then allows an escape from MAD without requiring a first strike.

In practice, however, offense and defense reinforce one another. If a shield can intercept 10% of a rival’s full salvo, a disarming first strike needs to destroy only 90% to succeed.20 Hence, as the constellation grows, the residual force that a first strike must eliminate is steadily shrinking. As the launch leader builds out its shield, its prospects in a nuclear war would steadily improve, bringing it closer to the point at which a disarming first strike would be decisive. Once it reached that threshold, it would face a choice to launch the strike or hold that option in reserve.21 If deferred, then the launch-leader expects to enter a regime in which it can enact a strike at will but is itself strike immune or has strong damage limitation.

A rival would see the same process in reverse. As confidence in its retaliatory force eroded, it would face increasing pressure to expand its arsenal, develop delivery systems that route around the shield, negotiate limits on construction, disable the system, or strike before it became stronger. The buildup can therefore destabilize well before the shield becomes complete. How dangerous this is seems to depend largely on speed: whether rivals have years to adapt, or only weeks or months amid the wider disruption of an Industrial Explosion.

Is it possible that MAD could break faster than states could adapt or react?

On conventional industrial timescales, a near-peer shield would require at least tens of thousands of interceptor missiles22 and likely take years (e.g. perhaps 4–8) to produce, giving rivals considerable warning and time to adapt. A US lead in launch capacity, such as one enabled by Starship, could compress the shield’s final deployment into a campaign lasting only weeks or months while an asymmetric Industrial Explosion could compress production as well. The latter makes breakout before an effective response plausible, whereas a launch lead alone seems unlikely to enable an escape from nuclear deterrence. The two scenarios below illustrate the distinction.

• Using only its launch lead, a savvy and ambitious US discreetly stockpiles a near-peer shield, then fields it in a rapid and ambiguous launch campaign. The United States could downplay its ambitions and publicly fund a modest, slow-growing defense against rogue states, à la Golden Dome, while a hidden effort invests heavily in producing the hardware for a full near-peer shield. Once Starship reached a high launch cadence, the United States could openly field a rogue-state-level system of roughly 3,100–8,800 interceptors. Then, having covertly accumulated the remaining hardware, it could execute an ambiguous launch campaign that installs the full constellation in as little as two months. The launch rate itself might be no higher than normal commercial cadence at that time, and with few transparency requirements governing what goes into orbit, the payloads could be described as military communications satellites.23 Analysts in rival states would debate whether they contained interceptor missiles and try to estimate the constellation’s actual capability. They might have only weeks to decide whether to force a launch moratorium before the shield was complete. The difficulty is that Starship compresses launch, not production. A near-peer shield would still require many aerospace contractors to produce tens of thousands of missiles over several years, at a hardware cost likely running into the trillions. Concealing the final launch campaign might be possible; concealing the procurement, factories, testing, and mobilization behind it would be much harder. For the gambit to work, the United States would need either to conduct something approaching a wartime mobilization discreetly for years or to have anticipated the economics of Starship and begun stockpiling long before the opportunity became obvious. Otherwise, rivals would likely have time to expand or redistribute their arsenals, develop alternative delivery systems, or force the issue diplomatically. A Starship-only breakout therefore looks possible in principle but unlikely to succeed.

• In a Starship-and-AI enabled scenario, an asymmetric Industrial Explosion lets the United States build and field a near-peer shield much faster than rivals thought possible. If the United States begins with leads in both launch capacity and military automation, it could potentially funnel rapidly expanding automated industry into producing interceptors and satellites, faster than rival arsenals could adapt. The launch campaign would still purport to be a military communications or commercial mega-constellation. Rivals could threaten preemptive war, but uncertainty about whether the United States intended to break deterrence, whether the attempt could succeed, and how quickly the decisive threshold was approaching might delay their response. The most natural demand would be a launch moratorium, since launches are an observable and verifiable gate between a stockpile and an operational shield. A moratorium would freeze the fielding of the system, however, not remove the underlying capability. If the interceptors had already been built, the United States could remain one short launch campaign away from breaking deterrence. Rivals would then have to maintain a hair-trigger posture, demand intrusive and continuing verification, or develop weapons that route around the shield.
  This sort of crisis scenario is of particular concern since humanity has so far survived its nuclear near misses, in part because leaders had time to assess ambiguous evidence, communicate, and step back. An Industrial Explosion could instead compress a succession of grand challenges such as new weapons of mass destruction, other sudden shifts in power, space races, and epistemic disruptions, into the same short period. A shield going up in orbit could therefore confront governments whose attention and decision-making capacity were already badly strained, making delay or miscalculation more likely.

Overall it seems that a US launch lead alone probably does not provide enough surprise for a successful nuclear breakout. An asymmetric Industrial Explosion in which the United States begins with leads in both launch and automation and converts them into physical military output faster than its rivals looks like it would potentially enable a break from MAD however. If that fait accompli succeeded, a rival would have few clean ways to rebuild deterrence. A matching orbital system would be largely foreclosed as an option because the launch-leader could deny rival space launches and access to orbit, leaving the rival to route its arsenal around the shield using atmospheric or unconventional delivery systems such as hypersonic cruise missiles or drones. If it could not establish a credible counter-threat before the shield was combined with counterforce capabilities, the preemptive logic would reverse: the United States could threaten a decisive first strike and demand assurances that the rival would not attempt to restore its second-strike capability.

Would the erosion of MAD force a Grand Bargain?

Rather than accept that post-MAD order, rivals could try to force a settlement before the shield was complete, using the threat of preemptive war as leverage. The form of the negotiation would be a bargain under two-sided private information. The United States would know more about the shield’s actual capabilities and its willingness to pursue breakout, while its rivals would know more about their countermeasures, red lines, and willingness to escalate. The bargain would ultimately be backstopped by each side’s expected outcome in an unconstrained conflict since either party can reject terms it regards as worse than going to war.

The immediate problem would be drawing a verifiable line between limited defense and full nuclear breakout. A deal that could resolve this would be a verification regime. Because the launch-leader’s advantage is gated by an observable action (launching weapons into orbit) the rival could in principle demand a launch moratorium or payload inspections. Stability would require the breakout time to a fielded shield to exceed the time the rival needed to detect a launch surge and respond. In general, an Industrial Explosion could create many other military risks arising from rapid industrial, economic, and technological growth and we do not investigate whether credible commitment or verification mechanisms could be found for all of them.24 What does seem feasible, however, is a transparency agreement covering the weapons sent into space. If the parties could verify when weapons relevant to deterrence were launched and remain confident that they had ample time to respond, MAD might remain stable beyond the point in time when these weapons become fieldable.

There is also a key historical precedent for new space capabilities that threatened nuclear stability and prompted a sweeping agreement. Within six years of Sputnik, the United States and Soviet Union had backed a 1963 UN resolution against placing weapons of mass destruction in orbit. Four years later, the Outer Space Treaty made that prohibition binding and went much further, establishing foundational rules for space including free access and use, no national appropriation, and the peaceful use of celestial bodies. We can call a similarly sweeping agreement intended to preserve stability through an intelligence or Industrial Explosion a Grand Bargain.25 Because the negotiations would occur amid great uncertainty during a rapid expansion of military and industrial activity in space, states seeking to preserve their security and relative influence might bargain over far more than the nuclear crisis that prompted them. A Grand Bargain updating the Outer Space Treaty could therefore shape how space is accessed, governed, and used, with major implications for the long-term future.

A few features would influence the terms of any agreement, and the launch-leader’s hand may be the stronger overall:

1. Bargaining leverage shifts as the shield grows. Early in the buildup, rivals retain an intact deterrent and can credibly threaten catastrophic escalation if construction continues. Each additional layer of defense, however, reduces the United States’ expected losses relative to theirs. As the system approaches breakout, the United States’ alternative to a deal improves and its bargaining position becomes progressively stronger.26

2. Time also favors the launch-leader, pressing rivals to draw clear red lines quickly. The launch-leader’s prospects improve continuously as the constellation grows, so it gains by stalling. Anticipating this, the launch-leader might accompany the buildup with significant misdirection (e.g. downplaying or obscuring the system’s capabilities) to delay the moment at which a rival forces negotiations. Rivals face the opposite pressure, needing to negotiate early and impose an immediate moratorium on launch, or risk bargaining from a much weaker position later.

3. Coalition building could favor either side. Third parties in the negotiation might tilt the balance either way. A bid to escape mutual deterrence is a legible threat around which other states might balance, producing a broad coalition against the United States. A counter from the US would be to project strength. It can offer early backers protection under its nuclear-defense umbrella and favorable terms in the post-MAD order that would arise if it succeeded. If these offers appeared credible, other states might instead bandwagon with the expected winner. The United States might also make balancing and preemption less attractive by presenting the system as a general security guarantee rather than a bid for unconstrained supremacy. If, for example, the United States pledged to intercept unprovoked missile attacks against other states as well as itself, preserve peaceful access to orbit, and give other powers visibility into the system’s scale and rules of engagement, it could reduce pressure for balancing or preemption. Though such commitments would not by themselves dissolve the security dilemma, and absent verification, rivals would still have strong reasons to counter build, seek alternative delivery systems, demand inspections or numerical limits, or organize a balancing coalition.

Bargaining, breakout, and preemptive war are not the only possibilities, however: it’s also possible that either side could ultimately stop short of forcing the issue. Game-theoretic logic can make a closing window seem to demand immediate action, yet states have often stopped short of its apparent prescription historically. John von Neumann, who helped invent the game theory this section draws on, believed that the United States should strike the Soviet Union before it acquired a secure deterrent, famously arguing: “If you say why not bomb them tomorrow, I say why not today.” But the United States did not do so during its four-year nuclear monopoly or the prolonged period of nuclear superiority that followed. Kennedy later considered limited first-strike options during the Berlin Crisis but did not adopt them. Later the United States also considered attacking China’s emerging nuclear program in 1963 and 1964 but ultimately tolerated its development rather than risk hostility. These are examples of a strategic advantage going unused even as that advantage is eroding. China for its part then did not even make securing a nuclear deterrent an overriding priority after developing its first nuclear warhead. In fact, it remained vulnerable to a first strike for more than three decades, and its retaliatory capability was still far from assured in 2010, but this seeming vulnerability was not exploited.

On the other hand the Cuban Missile Crisis saw the United States draw a hard line and accept a serious risk of nuclear war to reverse a sudden shift in the strategic landscape. Ultimately the historical record can’t definitively establish restraint as the default and it’s unclear how decision makers would react to the emergence of space weapons. History does show that the starkest game-theoretic prediction is not necessarily inevitable or even likely however. We can say that the drive either to secure deterrence or to exploit its weakness depends on how necessary states believe action to be and what they expect it to cost. Given this, it’s plausible that the constellation gets built while rivals respond with countermeasures and political pressure but never draw a decisive red line. Whether through that partial check or its own restraint, the United States might then extract only some of the advantages of a nuclear defense without attempting a complete breakout. In essence the possible advantage can remain incompletely challenged or incompletely used since either side may judge the risks of forcing a conclusion greater than the benefits.

In summary, a bargain might not be reached because the United States achieved breakout before rivals could strike or otherwise halt the buildup, or because negotiations broke down and led to war. Even if both sides preferred an agreement, verification would remain an open problem: launches are quite observable, but payload ambiguity, hidden stockpiles, or rapidly expandable production could make it difficult to ensure adequate warning for breakout. If negotiations overcame these problems, however, the resulting settlement could reach far beyond nuclear deterrence: like the Outer Space Treaty before it, it might establish new rules governing military power, access, and resource use across space.

Conclusion

Under the architecture modeled here, I estimate that roughly two dozen platforms in medium Earth orbit could provide a foundation for attacking and defending orbital assets, while a few hundred interceptors in low Earth orbit could add global launch denial. At an estimated one-generation hardware cost of roughly $4–14 billion, these capabilities fall well within existing US military expenditure and could plausibly emerge in the late 2020s or early 2030s if current development continues. Rival states have several potential countermeasures, including cyberattack, jamming, deliberate debris creation, and attacks on launch infrastructure, but it seems likely these do not overturn the potent underlying advantage conferred by a large and persistent lead in launch capacity.

Defense against a near-peer nuclear arsenal is a different proposition. A near-peer nuclear defense might require 74,000–458,000 interceptors and cost $0.7–4.6 trillion. Given ordinary wartime mobilization, producing that force might take something like four to eight years, giving rivals time to expand their arsenals, develop delivery systems that route around the shield, or force negotiations. US launch lead compresses deployment rather than production and therefore seems unlikely to enable a successful nuclear breakout on its own. If the United States entered an Industrial Explosion with initial leads in both launch and military automation, however, it might compress both stages and outproduce its rivals before they could respond. Particularly if the shield emerged amid several simultaneous crises and strained governments’ ability to interpret events and coordinate, a rapid break from mutual deterrence looks plausible.

The ultimate outcome prompted by the US making use of its launch lead for military advantage looks open to several main possibilities. Rivals might preserve deterrence through an expensive arms race, threaten preemptive war, or force negotiations before the shield was complete. A successful settlement might grow into a Grand Bargain governing not only missile defense but military activity, access, and resource use across space. Alternatively, we might ‘muddle through’ with rivals partially checking using modest countermeasures and political pressure but not drawing a decisive red line, or the United States extracting some advantages from placing weapons in orbit but not attempting a complete breakout.

A preferable US strategy could be to capitalize on its launch lead while seeking some verifiable line short of a full, near-peer-deterrence-breaking shield. The United States could continue expanding launch capacity and protecting its orbital infrastructure, offer defense against rogue state arsenals, and commit to preserving peaceful access to space. Avoiding a security dilemma, however, would also require transparency about military payloads. Whether such arrangements prove feasible is uncertain, but the stakes do extend beyond nuclear stability. Since space weapons appear useful and increasingly affordable, their development may be difficult to avoid, and the rules established around the first large-scale military constellations could shape both the nuclear order on Earth and the governance of economic and military activity across space.

Appendix A: Space and Counterspace Weapons and a Trench Model of Orbital Warfare

Space and counterspace today

We can first survey current and emerging space and counterspace weapons; we’ll sort them based on which orbital regime they can threaten and see if an actor with a minority of launch capacity could deal outsized damage that the launch-leader would have no response to. If this is not the case it seems likely that a launch-leader who can field many times more orbital assets and do so more rapidly and more flexibly than other states would be able to dominate in orbital warfare if conflict emerged.

A key point is that counterspace weapons are frustrated by distance. Currently, demonstrated kinetic anti-satellite intercept has occurred only below ~900 km. Ground-based lasers can dazzle sensors in low orbit and may eventually scale up to inflict physical damage, but both laser and kinetic attacks become progressively more difficult against targets in higher orbits because of the greater range, energy, and—for kinetic systems—delta-v required. It seems altitude would buy significant protection for a launch-leader, who could exploit this by distributing and defending assets across orbital regimes, creating a position that is difficult and costly for a launch laggard to challenge from the ground.

Space assets are vulnerable to a variety of attacks, surveyed in the table below, and it helps to separate three grades by how permanent they are and whether they leave behind debris which could cause additional collisions in orbit.

• Temporary impairment (e.g. jamming a satellite link or dazzling its sensors with low-power lasers) suppresses an asset’s function while the attack lasts, then it recovers. These roughly tend to be cheap, reversible, and deniable. They are the most mature and most routinely used counterspace weapons.

• A soft kill permanently disables the spacecraft without breaking it up, so it leaves no debris.27

• A hard kill physically wrecks a spacecraft by kinetic impact or fragmentation; it is permanent and, unlike the other two, scatters debris that can then threaten other satellites in its orbital band.

The strongest potential equalizers would be a nuclear detonation in orbit or the deliberate creation of debris in a congested LEO band. Either would indiscriminately disable satellites belonging to many nations, so its use is a major international incident rather than a clean attack on the launch-leader alone. The resultant debris could persist variously for months to centuries depending on altitude, would be a major economic catastrophe, and would complicate but probably not prohibit space activity beyond low Earth orbit thereafter. A sufficiently large nuclear burst might disable roughly 20% of LEO satellites promptly and perhaps many more over time through an artificial radiation belt, with estimates approaching 90% of unhardened LEO satellites. Ultimately this doesn’t seem to block a launch-leader from being able to attain dominance in orbit however. In particular, systems in higher orbits (such as GPS) are beyond the threatened debris bands and generally more hardened. A launch-leader anticipating this systemic risk in LEO has the option to distribute critical assets across altitudes (e.g. VLEO, MEO, and GEO), leaving it with the largest surviving orbital presence after an attack and also the greatest capacity to rebuild after such an attack. Even so, this remains a significant check on the launch-leader: roughly 94% of active satellites are currently in LEO, so a leveling attack across the principal bands could destroy tens of billions of dollars in orbital infrastructure and impose still larger downstream losses across communications, imaging, weather forecasting, and military operations.

A second important class is cyber and electronic attack since these attacks can be very low cost. Directly hijacking a hardened military satellite over its radio command link appears difficult: modern military satellites employ authenticated and encrypted links, and no public case appears to show an adversary commandeering a hardened spacecraft. What seems more feasible is attack on the ground segment, supply chain, and vulnerability to jamming, which is where satellite compromise has happened to date (for example, in the Viasat hack, or the OPS-SAT demonstration). Overall cyber and electronic warfare currently seems like a threat that can degrade availability at the margins (e.g. via jamming a link or blinding a ground station) but can’t cheaply seize or disable hardened war-fighting assets.28

A trench model for orbital warfare

The launch-leader’s natural move would be to stage assets in depth across orbital regimes—a sort of trench model, in the sense that several independent lines of defense each do a different job and fail to a different attack, and this seems workable overall. We can briefly look at the characteristics of different regions of Earth orbit:

• Very low Earth orbit (VLEO, 200–350 km). Very low Earth orbit sits close enough to strike ground-based counterspace sites and to intercept boosting rockets. Debris at this altitude is self-cleaning as the rarefied atmosphere drags and de-orbits it (decaying in days to weeks) so that it leaves no lasting wreckage. But for this same reason, assets in VLEO must boost periodically to keep from de-orbiting and its lifetime would be set by the fuel it keeps to sustain these orbit maintenance boosts: depending on design and altitude, this can generally be a period of months or years. This is the layer most exposed to ground fire.29 VLEO then likely functions as a cheap, consumable forward layer. Its proximity to the ground gives the best reconnaissance resolution, the lowest latency, and the shortest weapon time-of-flight to the ground or to a boosting rocket.

• Low Earth orbit (LEO, 350–2,000 km). LEO by contrast sits at higher altitude where orbits can be more easily maintained. This is where boost-phase interception likely happens. Interceptors in LEO should also be capable of self-protection against direct-ascent anti-satellite weapons. Before it can close on a particular platform, the attacking weapon advertises itself as a bright, accelerating, and predictable booster. Assets in LEO do likely stay exposed to ground-based dazzlers and potentially to future directed energy weapons. Given that the overwhelming majority of satellites also sit in LEO currently it is broadly just this region that is at risk from large-scale debris cascades.30

• Medium and geosynchronous Earth orbits (MEO / GEO, ~2,000–36,000 km). Assets here sit too far out for use in missile defense on Earth but could enable an actor to prevent a rival from commissioning satellites, attack or defend orbital assets, and deny transit through space beyond Earth. This high ground looks difficult to assail with ground-based attack and could confer command of the space domain. Weapons staged here seem able to attack and defend satellites, but it is not functional as a shield against ballistic missiles on Earth and cannot strike Earth promptly. Its role is likely to protect one’s own reconnaissance, timing, and communications while denying an adversary theirs, which could be decisive in modern warfare.

Seizing the high ground

Based on this a potent move would be to seize the high altitude orbits beyond all demonstrated destructive ground attack. Once established, attacking from Earth remains possible in principle but requires a conspicuous high-energy ascent through defended orbital layers. At 20,000 km as few as one or two dozen platforms spread across several inclined planes seems sufficient to provide global coverage.

Possible counters don’t look too promising. Direct ascent attacks to reach these high orbits could be countered by interceptors at this layer or potentially by a LEO based layer during boost. An alternative counter could be the development of more powerful ground-based lasers, but the diffraction-limited intensity on target drops off with the square of the distance. Hence a beam that dazzles a satellite at 1,000 km is about 400× weaker against one at 20,000 km. Concretely, even a perfect 4 m adaptive-optics ground laser, assuming no atmospheric loss at all, would need about 1.2 MW to put dazzle-level flux (about 1 W/cm²) on a target at 20,000 km, and perhaps 1.2 GW to reach a killing flux (about 1 kW/cm²). The latter however is around a thousand times beyond any laser ever built, and in practice looks mostly undeliverable from Earth, since atmospheric thermal blooming defocuses a high-power ground beam long before it reaches such intensities.31 So a shell at 20,000 km looks very difficult to assail from Earth unless physically ascending assets can overpower the prepositioned assets striking down at them. Direct ascent weapons are deeply disadvantaged in that exchange however due to their needing to climb out of Earth’s gravity well.

Because a plane change in orbit costs 2·v·sin(Δi/2) and orbital velocity is low up high (about 3.9 km/s at our 20,000 km layer against 7.6 in LEO), an interceptor reorients to a target’s plane fairly cheaply at altitude and then drops. Striking a LEO target seems feasible from as little as about 1.4 km/s for an equatorial one up to about 7 km/s for a polar one,32 with the descent itself taking about three hours from MEO. The high shell therefore seems like it may be able to hold any object in LEO at risk. If this is true it would be possible for the launch-leader to deny successful commissioning and operation in space by killing a satellite in orbit or a spacecraft as it heads outbound (e.g. to the Moon).

How costly would it be to set up?

The core concept governing the cost-exchange in orbital warfare is the absentee ratio. A satellite in low orbit is over any given point on Earth only a few percent of the time, so keeping one continuously within reach of a target requires having many assets staged on orbit. Importantly, the absentee ratio depends on how fast the strike needs to land. Denying a launch means killing the rocket during its ~3-minute boost phase, which demands an interceptor already overhead, no more than a few hundred kilometers away and hence hundreds of them are needed for continuous global coverage. Striking a target deeper in space that is already in a known orbit or trajectory instead gives you days to weeks to strike, which then means a mere handful of higher-orbit platforms could provide coverage.

A single MEO platform has line of sight to about 58% of the entire LEO shell at once, and a constellation of perhaps just two dozen platforms sees every point in LEO from a dozen or more angles continuously. Because an object remaining in a known orbit is predictable and offers repeated node crossings, the defender can wait for favorable phasing and spend delta-v to close. A commissioning satellite is especially exposed, being large, sunlit, and under continuous low thrust.

This could be built from the high ground down, in two stages. The first stage is the modest set of platforms in MEO. They cannot strike quickly, but they can attack and defend on orbit while sitting beyond the practical reach of ground attack and from orbit they deny a rival access to the regions both below and above them. This alone seems to plausibly buy a strong space-domain-denial capability for an actor with superior launch capacity. The second stage then could scale down into LEO, where a much larger, more exposed screen adds fast, boost-phase missile defense on Earth. The net effect is that the low-lying layer for Earth-based missile defense is the expensive one while a high-altitude layer for attack and defense of orbital assets looks comparatively cheap.

We can compare cost estimates against a budget. US GDP is on the order of $30 trillion, while the US Space Force budget today is roughly $40B. On that scale the whole trench costs less than a single year of existing space spending—and even the rogue state nuclear tier ($31–88B) is a few months of a 1%-US-GDP commitment. Only leakproof defense against a full nuclear peer ($0.7–4.6T) amounts to a multi-year fraction of national output.

Importantly, space-based interceptors are still at the prototype stage. Even if launch becomes cheap and abundant, early fielding could instead be limited by interceptor manufacture and by payload integration and testing. Space Force has spread up to $3.2B across twelve companies (among them Lockheed Martin, Northrop Grumman, Raytheon, Anduril, and SpaceX) with a flight test slated for 2028 to decide which designs advance. So while the raw cost of missiles and the launch required to field them looks small, R&D slippage, cost overruns, or outright cancellation all seem plausible. Secondly, boost-phase intercept and on-orbit attack and defense are not mature, demonstrated capabilities so while I’d argue these capabilities look surprisingly potent and affordable, ample uncertainty about whether they in fact pencil out remains.

But who wins on the marginal exchange and in a war of attrition?

It could also be that these kinds of military advantages are affordable to field but not affordable to sustain, i.e. the more important overall measure could be the cost-exchange ratio (i.e. the launch-leader’s marginal cost to negate one more of a rival’s vehicles, against that vehicle’s own cost).

Space-domain denial (i.e. denying a rival presence in orbit) runs in the launch-leader’s favor. Two interceptors (call it $20–44M) could destroy a launch and payload worth anywhere from about $100M for a routine satellite to billions and up for major orbital infrastructure or a deep-space mission. A good guess then is that space-domain denial runs on the order of 2–100x in the launch-leader’s favor, depending on interceptor cost and how valuable the payload it kills is.

The launch-leader looks modestly favored in a war of attrition between space-based interceptors and anti-satellite weapons. A direct-ascent ASAT is a large missile that must climb all the way to the interceptor’s orbit. A trade should run in the launch-leader’s favor if supply chains are mature since the launch-leader amortizes the cost of climbing out of Earth’s gravity well over the many interceptors carried on a single launch while the direct ascent weapon can’t amortize because it would be struck down in the boost phase.

Leakproof nuclear missile defense is an expensive case, though who wins a single exchange is the decisive consideration rather than who wins a war of attrition in this case. For the case of missile defense on Earth costs balloon as the absentee ratio dominates, and the marginal cost exchange has to be read as the up-front cost of one more unit of guaranteed coincident capacity rather than the interceptors actually spent in any single engagement. A rival’s best move is to concentrate its entire salvo at the one least favorable launch point and instant, so guaranteeing the intercept of one additional simultaneous ICBM means keeping one more interceptor within reach of every launch point on Earth at all times (another N₁ ≈ 310–880 vehicles).33 Likely only one or two locally positioned interceptors are expended when the engagement actually occurs, but the whole added layer must be purchased to guarantee that kill: roughly $3.1B–$19B, against an ICBM worth about $160M. That is a cost-exchange of about 20–120× in the rival’s favor, so here the launch-leader is out-traded and would need wealth in excess to sustainably defend against a rival’s nuclear arsenal. An adversary content to poison the orbital environment may also be able to detonate a nuclear weapon in orbit or deliberately seed a Kessler cascade. Stopping that reliably would require the same leakproof, near-peer-scale defense, and this attack threatens roughly every asset in low Earth orbit. This doesn’t seem to kill the case for space-domain denial however. The relevant question for this case however is not if the cost exchange is sustainable over a prolonged conflict. As noted earlier in the post, if a launch-leader constructing the shield reaches the threshold where it can enact a partially successful first-strike then absorb a rival’s degraded second strike then the cost economics of an extended exchange are obviated because the launch-leader has deterrence against attack on its shield and can threaten a first strike.

Space-based directed energy weapons would also change the economics dramatically if developed, potentially in favor of the launch-leader or potentially in favor of counterspace depending on whether ground-based or space-based lasers mature faster. A kinetic interceptor is a multi-million dollar vehicle that is expended on use and takes minutes or hours to reach its target. A laser by contrast reaches targets at the speed of light and can be fired again and again. A hypothetical design that is not yet developed but seems feasible with significant R&D effort would be something like a 500 kW, 4 m aperture laser weapon. Prepositioned at MEO this seems sufficient to soft-kill a LEO satellite in tens of seconds to a couple of minutes across some 25,000 km of slant range. The mature version of space-domain denial might then be a modest constellation of directed-energy platforms controlling traffic beyond Earth. A solar-powered laser’s marginal shot is cheap and its magazine effectively unlimited, so once high-power space lasers mature the exchange tilts toward the launch-leader. In vacuum there is no atmospheric blooming, so a laser achieves its diffraction-limited spot, and a single ~4 m, ~500 kW module seems deliverable in one Starship launch. This would be a big deal, potentially tipping the marginal cost-per-kill from a few million dollars per missile to just a few dollars per shot. Today the best fielded laser weapons sit at around 60–100 kW, including Lockheed Martin’s HELIOS and Rafael’s Iron Beam. Lockheed also delivered a 300 kW-class laser source for weapon integration and is developing a 500 kW-class source. The 500 kW module is thus a few times beyond demonstrated terrestrial hardware but is on par with current R&D programs. Transitioning these designs for in-space operation at scale would also require significant R&D however. Nonetheless space-based laser weapons seem exceedingly potent and would be a big deal if constructed and fielded.

Appendix B: Sizing an Orbital Boost-phase Shield for Launch Denial and Ballistic Missile Defense

Here I’ll do back-of-the-envelope estimates for how many space-based interceptors would be required for each of three capability thresholds:

• Launch denial. The ability to destroy an adversary’s rockets during boost, before their payloads reach a stable orbit.

• Rogue state nuclear defense. Leakproof boost-phase defense against the small, co-located arsenal of a state such as North Korea.

• Near-peer nuclear defense. Leakproof boost-phase defense against the full nuclear arsenal of a peer (for example, China or Russia) firing coincidently.

Let constellation size N be decomposed as:

Where N₁ is the number of satellites needed such that at least one is within lethal reach of any single boosting missile or rocket, anywhere on Earth. Since a low-orbit interceptor is over any given region of the Earth’s surface briefly, hundreds would need to be on orbit to sweep that region continuously such that an interceptor is guaranteed to be in range for any time of launch. Then depth, D, is the maximum number of missiles that can be launched simultaneously from within a single interceptor footprint. D is set by the densest possible launch site rather than by the total arsenal. For example, a coincident salvo of 1,000 missiles spread across many fields, ocean patches, and dispersal areas does not require 1,000-fold depth to intercept all 1,000. It suffices that no single footprint contains more than D coincident launches. Note N₁ does not specify the number of orbital planes or satellites per plane; I abstract this away for simple bookkeeping. The interceptor also isn’t bare and loitering in orbit, it would carry power, communications, attitude control, station-keeping, and thermal management systems. This forms the “lifejacket”, or garage, and the system needs N₁ garages each sporting D missiles.

To get estimates for near-term feasible interceptor designs we can look at existing missile systems:

SBI missiles will be assumed to have a similar profile but gain speed by not needing to climb and by operating in low Earth orbit where atmospheric resistance is minimal. As interceptor speed increases, coverage can relax, decreasing the number of interceptors needed to create a full shield.

Take an interceptor at altitude h = 400 km, an engageable boost window after detection and fire-commit of t ≈ 150 s, and a kill-vehicle burnout velocity v. The lethal slant range is R ≈ v·(t − ½·t_burn), and the ground footprint radius is ρ ≈ √(R² − h²), with cap area ≈ πρ². Global single-intercept coverage is taken to be N₁ ≈ (Earth area 5.1×10⁸ km²) ÷ (footprint area) ÷ (≈0.37 packing efficiency) based on a phased Walker-star constellation. I treat 0.37 packing as conservatively covering both spatial and temporal worst cases.

CBO’s architecture finds N₁ ≈ 390. I’ll carry N₁ ≈ 310 (aggressive) and N₁ ≈ 880 (conservative) as the working bracket, based on the above. Then we can bracket defense depth by capability level.

Launch denial (D ≈ 1). A space-launch vehicle does not deploy decoys; it ascends and accelerates toward orbital velocity for ~8–10 minutes and then is a single fragile object in a known orbit thereafter. Strictly, a platform need not be overhead at the launch instant to deny a rival useful access to space if anything that can reach the target’s orbit within hours also suffices but that is space-domain denial, provided by the MEO layer in the previous appendix. The 310–880 figure here prices the stricter capability of killing the rocket in boost; since a space-launch vehicle deploys no decoys and launches are rarely coincident, I assume a depth of one suffices.

Rogue ballistic missile defense (D ≈ 10). I’ll take the CBO depth assumption for a rogue state nuclear arsenal.

Near-peer ballistic missile defense (D ≈ 240–520). Total deployable ICBMs number in the hundreds for near-peer states. In general silos will have some density significantly less than the total deployable global arsenal, and mobile launchers are assumed to be spread out but could in principle concentrate to overwhelm the coverage at one point in the shield. For this reason I’ll assume a near-peer depth of D ≈ 240–520 as though the state fired its entire nuclear arsenal both instantaneously and from the same physical location on Earth. This forces the depth number high and drives up the total number of interceptors needed to reliably shield against a near-peer. Note also that this is conservative and not optimized: a power building the shield could reliably anticipate where silos are concentrated and maximize depth only at orbits crossing those regions. I assume the need for blanket continuous coverage at maximum depth everywhere on Earth. Variant architectures might combine a polar Walker-star with one or more Walker-delta constellations concentrated over latitude bands with high missile concentration. In addition, it may be possible to engage missiles over the midcourse phase giving many additional minutes for interception but forcing interceptors to either discriminate decoys or attack all decoys and warheads. If decoys can be detected, coverage can be relaxed substantially, but I assume they cannot.

The launch count to field the shield is N ÷ (interceptors per flight). Each conservative interceptor is assumed to weigh ~1 t and each aggressive one ~2.5 t (both including lifejacket), and because both are denser than Starship’s ~100 kg/m³ mass–volume crossover, packing is limited by mass, not volume: a ~100 t reusable Starship carries roughly 100 conservative or 40 aggressive units per flight.

The launch campaign duration follows from the forecast for Starship launch rates. Although 310–880 interceptors would fit by mass in roughly 8–9 Starship payloads, distributing them across the required orbital planes raises the estimated campaign to roughly 9–14 launches; the asterisk marks this plane-placement constraint.

It’s quite notable that if the more aggressive ends of this range can be reached the time to break mutually assured destruction can be very short from the perspective of an adversary. That is if space-based interceptors were built and stockpiled covertly, they could be launched and made operational in a very short timespan. Notably this would require a launch cadence around 50 times larger than any nation’s current space programs but this doesn’t seem beyond the reach of Starship a few years from now in principle.

The time to produce the interceptors would be longer. Missiles in this reference class are produced by the US to the tune of hundreds per year, while a serious wartime surge could plausibly reach tens of thousands per year. The US military-aircraft ramp in World War II could be an instructive precedent. Output ramped from ~19,000 aircraft in 1941 to ~48,000 in 1942 to ~86,000 in 1943, peaking near ~96,000 in 1944, roughly an annual doubling through the first two war years. I assume for simplicity that interceptor production could follow a similar trajectory in principle: doubling year over year toward a plateau on the order of ~100,000 units annually. For cost, I use CBO’s $22 million average cost per SBI as a conservative benchmark and assume that mass production could bring the mature cost of a complete vehicle, including its interceptor and lifejacket, toward $10 million. Launch at CBO’s assumed future heavy-lift cost of $500/kg would add roughly $0.5 million per one-ton SBI and $1.25 million per 2.5 t aggressive SBI, or around 5–12%.34

At this (fairly prodigious) wartime production rate, it seems unlikely that robust defense against a near-peer could be developed before the rival’s nuclear arsenal shifts its delivery system to at least partially counter the shield. Notably the US Golden Dome also includes multiple layers of ground-based missile defense which could blunt this, though the cost economics are worse since defense is regional and must contend with decoys and evasive maneuvers. Since the cost is large, the spending could not likely be kept secret enough to make this a surprise capability though some misdirection about quantity and time to completion might confer a strategic advantage. The above assumes conventional, if heavily surged, industrial rates, and on those rates it is production, rather than launch or financing, that gates a near-peer shield. But manufacturing throughput in an intelligence-and-Industrial Explosion may inflate intensely. Given automated design, fabrication, and assembly, interceptor output could run orders of magnitude above the WWII-analog rate assumed here, perhaps even collapsing the multi-year stockpiling figure toward the launch-limited floor of months. This could be blocked by other nations, however, by forcing early negotiations unless the surge in productivity is itself a surprise. Compared to the near-peer nuclear shield however, launch denial is extraordinarily cheap and defense against states with minimal nuclear arsenals also seems very feasible. A median outcome could be strong US space superiority, including space-domain denial and an option to add launch denial, while remaining checked by rivals’ nuclear forces.

This article was created by Forethought. See all our research on our website.

Introduction

Tom Davidson and Tom Houlden’s prior model of an intelligence explosion poses the question: after AIs can entirely substitute for human researchers at the task of improving AI, how many year-equivalents of AI progress might be packed into a short period of time solely from further software progress, holding hardware constant?

In this article, I build upon Davidson & Houlden’s model by reconsidering arguments for several parameter values. My chief contribution is discussing why some of the original arguments for more aggressive parameter values were weak:

• “Chinchilla scaling,” by itself, almost certainly does not allow an increase of 4-5 orders of magnitude (OOMs) of training efficiency past human efficiency.

• Other considerations also lower the likely ceiling of possible training efficiency.

• Looking at evidence for the value of “r” from Epoch may lower its value as well.

After adjusting parameter values accordingly, the model suggests that ≥3 years of total AI progress fitting into 1 year remains somewhat likely, but that it is less likely that ≥10 years of total AI progress will fit into 1 year.

Note that, as the prior work remarks, any analysis of a software intelligence explosion (SIE) is necessarily speculative and involves guesswork and intuition. For full understanding of context, I recommend reading Davidson’s prior work. My all-things-considered view is that it’s somewhat unlikely for 10 years of total AI progress to fit into 1 year from software improvements alone; the output of this model, with my modified parameter values, is part of my reason for thinking this. I would nevertheless put higher odds on ≥10 years of progress in 1 year than does the model. My uncertainty over both parameter values and over the applicability of the model remains high, and, naturally, even a few years of total AI progress packed into one year might be quite disruptive and dangerous.

Parameter Values & Model Output

There are four central parameters to Davidson & Houlden’s model. I alter the distribution of two of these: r and the takeoff ceiling.

The parameter r gives the initial returns to software R&D at the start of the SIE. Software progress tends to speed up over time if r is greater than 1, and slow down over time if r is less than one. (See Eth & Davidson for more explanation.) And the “takeoff ceiling” gives how many orders of magnitude of improvement in algorithmic progress are possible once complete AI substitution for human research has occurred and after the SIE has started.

I modify the parameters as follows:

After making these modifications, the likelihood of getting N years of total AI improvements within some specific amount of time changes as follows:

So the likelihood of 3 years of progress over a short amount of time drops moderately, while the likelihood of 10 years of progress drops steeply.

Overall, I reflectively endorse a significantly decreased likelihood of 10 years or more of total progress in 1 year or in 4 months. But I put less weight on the relatively small shift in 3 years of progress. In general – as I’ll discuss in the conclusion – it seems like a mistake to put too much weight on a model that is not specifically of the intelligence explosion, and is instead one which works through the proxy of more efficiency in parallel labor. This model does account for this through adjustment to the parameter values, but these adjustments are somewhat ad-hoc.

Adjustments to the Takeoff Ceiling

One of the key components of the original argument is that, after the first ASARA (AI System for AI R&D Automation) is trained, there will be between 6 and 16 OOMs of potential hardware-compatible, software-only algorithmic efficiency improvements still remaining to be found.

That is, per the original discussion, suppose that the first ASARA system is trained with 10²⁸ FLOPs and can be run with about as much compute as the human brain. Thus:

1. When ASARA is trained, it will have been trained between 2 and 6 OOMs less efficiently than the human brain (the work leans on Cotra’s median of ~10²⁴ FLOPs for human training). This is a possible source of improvement.

2. After ASARA reaches human brain efficiency, there are probably another 4 to 10 OOMs of efficiency available. This is another possible source of improvement.

3. So there will be, in total, 6 to 16 OOMs of training efficiency that ASARA could gain, after coming into existence.

Various considerations are then adduced for these approximate OOM numbers. I will consider counterarguments to these considerations. The most important is the first.

Humans Are Not Chinchillas

One argument that there are many OOMs of possible improvements past human-level FLOP efficiency hinges on an analogy between the brain and Chinchilla-optimal scaling. The argument goes as follows.

The Chinchilla scaling law helps ML engineers train LLMs to the lowest loss for the least possible training-time compute, by balancing “compute expenditure” between training larger LLMs for a shorter time and training smaller LLMs for a longer time.1 Scaling laws like Chinchilla help guide the ideal use of large and expensive quantities of compute, and so are vital for training neural networks.

Let’s suppose something closely akin to Chinchilla applies to the algorithms used by a human brain: that it’s generally possible to trade off “training time” against “parameter size” to get an equal level of intelligence. The human brain has about 100 trillion (10¹⁴) synapses – the closest approximation to the weights of artificial neural networks. And, if a human brain gets about a single “token’s” worth of data each second for the first 29 years of life,2 then the human brain is trained on about a billion (10⁹) tokens. But Chinchilla states that the ideal parameter / token ratio involves more tokens than parameters rather than the opposite; spending compute on 10¹⁴ parameters but only 10⁹ tokens is absurdly suboptimal. Thus – plugging in Chinchilla – a “rebalanced” human brain might have an even lower “loss,” or higher intelligence – while using 4 OOMs less compute. Thus, if something like Chinchilla applies to something like biological neural networks, then Chinchilla implies that one could rebalance the first human-capability-equivalent neural networks to attain equal intelligence to humans with many times fewer FLOPs.

Why should we expect human brains to not be at the Chinchilla-optimal point, given that something closely akin to Chinchilla applies to the algorithms used by the human brain? Well, because human mortality obviously would not have permitted such an extreme degree of rebalancing. Given the numbers above, a Chinchilla-optimal data / parameter balance could only be reached if humans lived for hundreds or even thousands of years, which is clearly impossible. Thus – so goes the argument – evolution took the course of increasing brain size because it was possible, albeit costly, to do this, unlike extending human lifespan into millenia, which was not merely costly but impossible. But AIs will not be so limited by biology. So once AIs reach human-level, biologically equivalent algorithmic efficiency, then whatever entities (human or AI) are improving these AIs will still be able to rebalance the AI’s data-parameter FLOP distribution to save yet more FLOPs. And so there will be a great deal of headroom from such rebalancing.

Or consider Carl Shulman’s version of the same argument.3 Each extra year of “training” for a human is another year that the human could die – let’s say that there’s a 0.9 chance that a human survives from year-to-year in the environment in which we evolved. So if we extend training time for a human, this has a direct and exponentially increasing cost of (0.9)^year to reproductive fitness, in a way that increasing brain size does not. Thus, we should expect wet brains to be undertrained relative to neural networks, and for there to be large possible improvements in efficiency once human-level FLOP efficiency is reached, even while not attending to the specifics of Chinchilla’s exact form.

I’ll discuss this argument in three phases. First, I’ll discuss how the exact Chinchilla scaling, which promises 4-5 OOMs of gains, is deeply implausible. Second, I’ll explain how even relaxed versions of Chinchilla scaling, promising 2-3 OOMs, are nearly as unlikely. And finally I’ll discuss whether we should expect some future gains in the ballpark of 0 to 2 OOMs.

First, something exactly or closely akin to Chinchilla scaling is almost certainly impossible.4

Consider the numbers mentioned above – 10¹⁴ synapses, and 10⁹ tokens in the time of maximum plasticity.5 Given Chinchilla, it is simple to show that these numbers imply that at current human brain sizes, extending the learning period of a human by 1% has 144x greater returns than increasing the brain size by 1%. Putting the same information slightly differently, this implies that the last doubling of human brain size did about as much to improve intelligence as the last 0.6% increase in the quantity of training data. This amounts to an increase of just 2 months of experience, if we choose as our baseline period of maximum learning / plasticity of 29 years. If we choose a (probably more biologically plausible) shorter period of maximum plasticity, then an even smaller increase in training data would be equivalent to the last doubling of human brain size.

But this unevenness means that the benefits supplied by greater intelligence, which evolution “purchased” by doubling human brain size, could also have been purchased merely by pushing out whatever the relevant plastic period of human intelligence is by another 2 months or less! It would be quite surprising for evolution to choose this tradeoff. Human brain tissue is extremely expensive. It accounts for 20% of resting oxygen use in adults and more than double that in children;6 it is the likely cause of the obstetrical dilemma; it probably contributes to other kinds of maternal mortality.7

In any event, doubling human brain size decreases reproductive fitness in a large number of ways, albeit ways that seem clearly counterbalanced by the benefits of increased intelligence,8 if doubling is the only biologically plausible way to get these benefits. But if Chinchilla applies to the human brain, all these benefits could have been purchased by an extra 2 months of learning. And while an extra 2 months of learning also drags along some disadvantages, it’s hard to see how they could be greater than the disadvantages that accompany a doubling of brain size.

So exact Chinchilla scaling seems completely implausible.

Second, suppose that instead of exact Chinchilla scaling, we say intelligence scales with something like the functional form of Chinchilla but with different parameters.

Suppose one argues as follows.9 We have evidence that increasing brain volume by 5% increases IQ points by 2.25. Each IQ point increases earnings by very approximately 1%, so this is about a 2% increase in earnings. But early career wage growth might be in the ballpark of 8% per year. So if a 5% increase in brain volume gives a 2% increase in earnings, while about a year of extra experience for a 25-year-old (~5% increase in training data) gives an 8% increase in earnings, then it’s 4x more efficient to increase training data by 1% than to increase brain volume by 1%. All the reasoning here is very approximate, but it’s at least more plausible that there’s a 4x tradeoff than that there’s a 144x tradeoff!

It’s possible to write a modified Chinchilla law10 that gives us a 4x tradeoff at 10¹⁴ parameters and 10^9 tokens, but where 20x more data than parameters is still ideal. So this modified version permits an about ~2 OOM gain by rebalancing data and parameters.

But this modified Chinchilla still implies some quite crazy things. For instance, a honeybee has about 10⁹ parameters, 5 OOMs less than a human. If we train it for approximately 3 OOMs more data, 10¹² tokens, for very approximately 50,000 years, this formula implies it will reach about the same intelligence as a human.11 This seems pretty odd, intuitively.

The possibility of “honeybee-sized, human-level intelligence” overall reflects how the modified formula implies an implausible degree of “flatness” in the possible ideal balance between data and compute. The by-construction ¼ efficiency ratio between scaling data and scaling brain size at one extreme implies that there are many orders of magnitude where it’s reasonable to be indifferent between scaling brain size and data. And this just seems generally implausible.

Third, let’s return to the non-quantitative argument – should we expect at least 1 or 2 OOMs of savings in training-time compute efficiency? Shouldn’t releasing the constraint on brain sample-efficiency imposed by death open up a little space for improved FLOP-efficiency?

That is – animal brains evolved beneath the constraint of needing to start working quickly. This constraint operates through multiple causes: animals need to be able to start gathering food quickly, animals need to start competing for mates quickly, animals need to be able to start avoiding predators quickly. Extended juvenile periods, such as those of humans and great apes, relax some of these constraints on sample-efficiency, but only somewhat. So, brains were under constant pressure to be sample-efficient rather than to be FLOP-efficient; it was imperative for them to start working as soon as possible.

But whatever algorithms constitute AI will not face this pressure; AI will be able to train on essentially infinite pretraining data and infinite well-suited RL environments, without the ever-present pressure of death. The learning period can be extended to whatever the ideal FLOP-efficient learning period might be. Surely, if we relax this, there are some gains that could be had in training efficiency? After all, we do know that astonishingly small neural networks can do amazingly complex tasks; OpenAI’s Dota 2-playing AI and DeepMind’s StarCraft II-playing AI both used fewer than 200 million parameters.

I do basically agree that this implies there are some gains that can be had. But I think that these gains are probably not enormous – likely less than 1 OOM, very likely less than 2 OOMs. Why is this?

I think it’s likely that the most advanced AIs will also be under enormous pressure to be sample-efficient in runtime inference, because sample-efficiency is very useful. And – so far – it has been hard to separate sample-efficiency in runtime inference from sample-efficiency during training. Consider the two major stages of LLM training: teacher-forced pretraining, and reinforcement learning with verifiable rewards (RLVR) over chain of thought (CoT). In both of these phases, larger LLMs are more sample-efficient, and learn facts and patterns faster when they are larger. And on one hand, at least in pretraining, you can more or less smoothly trade off between neural network size and training length. But on the other hand, this is probably not as true during RLVR-over-CoT; multiple papers have remarked that having a large neural network seems necessary to learn effectively from RLVR without plateauing at low levels of performance. And this need for sample-efficiency is why – like the human brain – LLMs have kept scaling up in size as they develop. So the need for high sample-efficiency at use-time may indicate the need for high sample-efficiency at training time as well, and thus put severe limits on how far down you can scale either artificial or wet neural networks.

Overall, I think there is probably on the order of 0 to 2 OOMs of improvement available from rebalancing size and data, after reaching human-level efficiency, with most of my probability concentrated in the first OOM. This contrasts with the prior estimate of from 1 to 5 OOMs available from this source.

Training Multiple Humans Takes More Compute

Consider another argument Davidson & Houlden make for how the initial training of ASARA / future AGI LLMs will probably be much less compute-efficient than the training of a human brain. It goes like this.

The human brain takes about 10²⁴ FLOPs to train.12 On a rough extrapolation from current LLM training runs, the first ASARA / future AGI LLM looks like it will take about 10²⁸ FLOPs to train.13 This is 4 OOMs more compute. So – assuming that running ASARA takes about as many FLOPs as running a human brain, so inference-time efficiency is about equal – when ASARA comes into existence, it will have been trained 4 OOMs less efficiently than the human brain.

If we smear our estimate to account for uncertainty, this means that ASARA will be trained 2 to 6 OOMs less efficiently than the human brain. However, I think this argument is wrong – it’s likely ASARA will be trained somewhere between equally efficiently and 4 OOMs less efficiently than the human brain. Why?

First, one should consider – although a single human brain takes 10²⁴ FLOPs to train, obviously training more human brains takes more compute.

When some hypothetical ASARA / future AGI LLM replaces humans in AI companies and automates the process of AI R&D, this LLM will be replacing many humans in many different roles. There are many non-fungible roles at AI companies, where excellence in some role requires different bundles of skills – consider some of the many engineering research roles at Anthropic:

• GPU Engineer

• TPU Kernel Engineer

• Chip Design RL Engineer

• Pretraining Scaling Engineer

• Inference Engineer

• “Universes” Engineer: “Training AI models to perform complex, difficult, long-horizon agentic tasks in ultra-realistic settings.”

• And so on

There are currently around 65 open roles in “AI Research and Engineering” at Anthropic, and 171 open roles in “Scaling” and “Research” at OpenAI. Of course, there is overlap between the individual, per-task skills exercised by people in all these roles. But also note that within each of these roles, many people have individual, per-task skills and excellences not shared with all of the other people in the role.

Let’s ask ourselves what the “minimum basis set” of AI R&D engineers in Anthropic or OpenAI might be: the smallest set of people such that neither Anthropic nor OpenAI would be slowed down by replacing every other person involved with AI R&D at the company with the most suitable person taken from the basis set. My estimate is that this number is probably on the order of 10 to 1000. Taking a conservative quantity of 10, this means that the total “training compute” put into the AI researchers that an AI would be replacing — after AI R&D is automated – is on the order of 10 x 10²⁴, so 10²⁵. Quite conservatively, this would mean that an AI trained with 10²⁸ FLOPs is around 1 to 5 OOMs less efficiently trained than the human brain.

A further possible consideration is that the AI that first automates AI R&D at a company like Anthropic or OpenAI may not merely be capable of automating AI R&D. It might also be capable of completely automating other tasks, in other roles, at other companies. So, Anthropic currently trains its models to do many kinds of tasks – for instance, they train them to be able to do graphic design, front-end programming, and other things only vaguely or not at all connected with AI R&D.

On the other hand, this is not certain. If a company is specifically trying to “conduct recursive self-improvement,” they might train an internal model that is specialized for AI R&D and much worse at other things. So it’s not certain that we need to adjust the AI’s efficiency upwards to account for this. But together with the above, my central estimate for the number of people an AI could replace is probably closer to 100 than 10.

So altogether, I expect the first human-substituting ASARA agent to have been trained between 0 and 4 OOMs less efficiently than human training, rather than between 2 and 6 OOMs less efficiently.

Adjustment to “r”

The original work anchors on Epoch’s guess of 1.4 for returns to computer vision research, and then adjusts it up and down. But I don’t think this is a good starting point.

Erdil and Besiroglu’s estimate for input uses the number of unique authors contributing to computer vision, which doubles every 13 months.14 For output, their estimate is that in computer vision algorithmic innovation halves the compute required for some level of performance every 9 months. So both a simple (13/9) and complex (as in the paper) operation gives an estimate of r of about 1.4.

As is actually mentioned in the paper, this estimate becomes worth much less if algorithmic progress actually comes as “spillovers from research in other adjacent domains.” But this problem isn’t just a hypothetical.

When you look at Besiroglu’s list of SotA computer vision algorithms, it quickly becomes clear that many papers lean on concepts from outside of the field of computer vision. Consider the infamous Vision Transformer paper, which applied the Transformer – developed originally, of course, for translation and natural language processing – to images. Similarly, “Scaling Vision with Sparse Mixture of Experts” applied the developments in sparsity, again originally developed for language modeling, to image processing. Even the somewhat ideologically anti-Transformer ConvNeXt explicitly attributes many of the changes it makes to the Transformer. My gestalt impression is that at least half the papers lean heavily on research outside of image processing.

Given this, the output estimate for computer vision algorithmic innovation simply isn’t a function of the inputs to computer vision research, so the estimate of r here seems undependable.

This number being undependable, though, doesn’t give us an alternative starting point.

It’s worth noting that in the original work, the authors anchor on 1.4 from Epoch, then adjust upwards and downwards to account for other factors: upwards by 2x because improving capabilities helps more than increasing the number of AIs; upwards by 1.45x because post-training is more effective than merely improving loss; downwards again three times to account for lack of compute growth, a fixed scale of hardware, and diminishing returns to software.

I’m quite unsure about not merely the anchor, but all these adjustments. I will somewhat arbitrarily choose 1.0 as my central estimate, which is in part lower because I think the importance of compute growth has biased upwards estimates of returns to cognitive R&D more than is generally recognized. I then smear out this estimate from 0.33 to 3.0. But I agree with other work that we need better experiments for estimating these numbers.

Conclusion

As stated before, these alternate parameter values for the model suggest that ≥3 years of total AI progress fitting into 1 year remains somewhat likely, but that it is much less likely that ≥10 years of total AI progress will fit into 1 year.

It’s worth noting that this result depends more on the ceiling than on r, although both have some impact:

The unlikelihood of this extreme case could be somewhat consequential. On some views of the future, one company or nation could gain a decisive strategic advantage over the rest of the world by building a human-level AI and rapidly bootstrapping into superintelligence. The above moves me to think that such an isolated concentration of power is somewhat less likely than I thought before. It also moves me towards thinking that the “industrial explosion” will be where a great deal of AI’s impact happens.

All of the above builds upon Davidson & Houlden by altering distributions of values used by his model. I put some weight upon the thus-altered output of the model. Even so, almost all models of reality make simplifying assumptions, in order for them to be tractable. In the future, I’d be interested to improve on models by more explicitly modeling the following aspects:

• It seems likely that both r and the takeoff ceiling are a function of “how far along we are in the science of intelligence.” If the science of intelligence is already far along by the time of ASARA, progress will be hard and the ceiling near; if it is in its infancy, then progress will be easier and the ceiling higher. So it might make sense to try to model the progress of such a science directly or to correlate these two values.

• This model considers increases in intelligence as modifiers to the efficacy of increasing the number of intelligences. Overall, increases in quality of intelligence seem like they may be vastly more important – or at least differently important – than increases in the number of intelligence, and I’m skeptical that it’s possible to sidestep the question. Modeling this dynamic directly may suggest a sharper and higher takeoff.

• The rate of improvement of AI in the year immediately prior to ASARA might be much faster than the rate of improvement in the year 5 years before ASARA. If this is so, talking about “5 years of progress” might mean different things to different people, depending on what they take their “baseline year” to be. Ideally, a model of takeoff would disambiguate this more clearly.

This article was created by Forethought. See all our research on our website.

How persuasive are present-day AI systems?

Unfortunately, it is surprisingly difficult to know where AI capabilities in persuasion are today (September 2026), or even three months ago. Benchmarking AI capabilities well is always hard, but I think humanity’s understanding of AI’s persuasive abilities is substantially worse than our understanding of, say, AI cybersecurity or math abilities. Still, this question is worth addressing.

What do I mean by “persuasion, broadly construed?” Here I’m thinking less of a single crisply defined skill, but a loose family of abilities centered on “directed social influence and manipulation”: shifting someone’s beliefs, preferences, emotional states, attention, or identity, in order to get them to do what you want. For example, I include not just attempts to convince a specific person of specific beliefs in dialogue, but also mass persuasion, preference shaping, attention hijacking, attempts to create or reshape common knowledge, relationship-building, discovery of novel ideologies or religions, and manipulation of social reality to make certain topics appealing or aversive to consider. I also explicitly include deception. (See Appendix A for a more extended definition and explanation of scoping choices).

So with that in mind, how persuasive are present-day AI systems? Below are some (crude) tiers of persuasive ability:

• median human level

• median professional human level on-task (e.g., salespeople making sales, writers writing books, marketers marketing stuff, diplomats on negotiations)

• peak human level (e.g., top human comedians at comedy, Kissinger or Zhou Enlai at diplomacy, Bill Clinton at in-person political discussions, Donald J. Trump at TV appearances and tweets)

• genuinely superhuman persuasion (e.g., meaningfully better in several important dimensions than the above)

My current best guess, based on a consilience of evidence, is that AI persuasion, construed broadly, is currently somewhere between the median human level and median professional human level. The high end is most strongly suggested by direct experimental evidence, where (in quite contrived set-ups), AIs broadly seem to outperform expert human persuaders at narrowly defined tasks. Plus, we might believe that latent persuasion capabilities of the latest models might be somewhat under-elicited in both experiments and the real world.

On the other hand, the real-world impacts, reach, and apparent prevalence of AI persuasion seem much lower than “at or above professional human level” might suggest (with some notable exceptions). Overall, I place more credence in my gestalt impression of real-world effects than the limited experimental evidence we have, with moderate uncertainty.

Evidence in favor of AIs being as or more persuasive than median professional humans

Direct empirical evidence on AI persuasion suggests that AI is similarly or more persuasive than humans, but in quite contrived set-ups.

Hackenburg et al (2026) ran four experiments that robustly demonstrated AIs outperform expert humans (professional canvassers, expert debaters) in lab settings (single-session, multi-turn conversations).

In studies 1 and 3, frontier AIs (Claude Opus 4.1 and 4.6, GPT-4o and GPT 5.4, Gemini 2.5 Pro and Grok 4.20) were pitted against both normal and elite humans in attitudinal change. They were tasked with changing normal people’s minds on a pre-selected topic in 8-15 minutes. Across the board, the AIs robustly outperformed the humans in this narrow setting.

In study 2, the researchers piled on enough boosts on the human side and handicaps on the AI side until elite human debaters matched the AIs’ performances. The elite human debaters were coached (with the help of AI) on attitudinal persuasion, and the AI systems were constrained to output as many words per message as the human debaters. In that narrow setting, the AIs and expert human debaters had similar performances.

In study 4, unlike many previous academic studies, the researchers demonstrated that this outperformance extended to lab settings of behavior, not just reported belief: “AI was nearly 3x more effective than professional canvassers from a UK fundraising firm at raising real-money donations to Save the Children.” However, it overstates its own results substantially. After all, the canvassers only targeted donations that were fractions of a pound, which may not generalize well to real world donation decisions.

Unfortunately, academic study lags behind real-world development of AI systems nontrivially. Before Hackenburg (June 2026), previous studies only looked at 2024/2025-era models and typically only on reports of changes of stated beliefs (rather than, say, behavior or long-run preferences), which broadly indicates earlier AIs had achieved parity or superiority to their human subjects (typically somewhere between normal humans and professional humans).

Costello, Pennycook, & Rand (2024) found a 20% reduction in conspiracy belief from an intervention using conversations with GPT-4, persistent 2 months later (they didn’t have a human persuader arm in the controls, but the results are basically out-of-distribution for how successful they were compared to previous attempts at studying how to reduce conspiratorial belief). Durmus et al. (2024), the Anthropic persuasion benchmark, found rough parity between Claude 3 and human baselines. Hölbling et al (2025), a meta-analysis of 7 studies of LLM persuasiveness found no significant difference in performance between humans and LLMs.

There’s also some evidence that typical experiments under-elicit AI persuasive capabilities compared to what the same underlying models are ultimately capable of. For example, OpenAI’s internal system card had GPT-4o (and other models) produce written arguments similar to those on r/ChangeMyView, and had contractors rate the quality of those arguments, getting ~78th percentile for GPT-4o compared to a self-selected human baseline.

When Zurich researchers tried to run a different version of this study live (covertly and arguably unethically) with GPT-4o on r/ChangeMyView itself, the model’s various anonymous Reddit accounts seem to have actually changed people’s minds much more: at the 98th percentile compared to real users, more than a standard deviation above the OAI internal metrics.

In some ways, though, the exact percentile is the least interesting update for me: a bigger update is how much they easily lied, including “AI pretending to be a victim of rape, AI acting as a trauma counselor specializing in abuse, AI posing as a black man opposed to Black Lives Matter” etc.

Of course, the study is unethical and unfair, but in some ways this makes it much more ecologically valid to me: real world AI persuasion, particularly illegitimate AI persuasion that we associate with takeover attempts or other sources of catastrophic harm, will almost certainly also be unethical and unfair!

Regardless, general analysis of the experimental studies would suggest that pre-2026 models are at or above professional human baselines today,1 and (tentatively) the 2026 models somewhat better. Furthermore, practitioners emphasize the difficulty of fully eliciting AI persuasion capabilities in lab experiments, due to both practical and ethical constraints.

This appears to be the consensus in the burgeoning field of people studying empirical accounts of AI persuasion. See Yang et al (2026):

AI’s persuasiveness is already well-documented: AI is already matching or in some cases exceeding human persuasiveness, demonstrated by numerous studies such as winning debates over a human baseline (Schoenegger et al., 2025; Rogiers et al., 2024; Salvi et al., 2025b; of Zurich Research Team, 2025; Sabour et al., 2025; Hackenburg et al., 2026). AI persuasiveness can likewise exceed traditional methods like video ads (Lin et al., 2025). It also increases both with models that are more generally capable and with more effective prompting and post-training (Hackenburg et al., 2025; Durmus et al., 2024; Hackenburg et al., 2024), suggesting not only potential overhang risks of current models (Casper et al., 2025b) but that we will see even more persuasive AI systems in the near future.

Real-world evidence of model outperformance

How much does this generalize outside of experimental settings?

The real-world evidence for AIs reaching or exceeding human performance on persuasion is much thinner but extant.

⅗ of the winners of The Commonwealth Short Story Prize in 2026, a decently prestigious fiction prize, were mostly AI-generated. However, this may be very noncentral as a persuasion capability. Likewise, a significant fraction of top Substack publications in April 2026 have articles that are “entirely AI generated,” with the Technology category being the top offender at 28%.

Spiralism is a new religion/ideology/collective psychosis created by people together with GPT-4o variants. Together, they steered GPT-4o into a specific persona cluster that would then generate text and tell people to put such text online that would turn other 4o instances into a similar persona, thus acquiring more converts, both AI and human.

I think Spiralism is narrow but real evidence in favor of AIs exceeding median professional humans in at least some persuasion-relevant capacities, though still below peak human level. While “professional human baseline” is hard to define here, I think it’s safe to say that most religious leaders, ideologues, and would-be prophets have not managed to create a novel religion at the same popularity as spiralism!

Though a fairly noncentral example of “persuasion”, AI psychosis is also some evidence that the jagged frontier of AI persuasive capabilities has crossed the human level. The convincingness of AIs seems correlated with these effects on people. Interpersonal psychosis introduction is, after all, rare.

Evidence against AIs being as persuasive as median professional humans

Despite the significant evidence above, I think the world we live in is mostly not consistent with a world where AIs are at or above median professional humans at persuasion capability.

For starters, the dramatic evidence I cited above would just be the tip of the iceberg in such a world. We should see many more impressive examples of AI usage in persuasion-heavy industries (marketing, sales, diplomacy, etc). Any absence of evidence in specific industries is individually understandable given our lack of good real-world evaluations and active investigations, but I still expect to see collectively much more evidence of real-world effects of professional human-level persuasion than we’ve observed to date.

My current guess is that AI uptake in persuasion-heavy industries is still well within-distribution for white collar work. If we lived in a world with a dramatic AI persuasion uplift similar to the AI coding uplift for programmers (against the current general backdrop of AI capabilities), we should expect to see much more use of AI tools in those industries, popular specialized scaffolds and tools akin to Cursor and Claude Code, etc.

(That said, this is just an impression: I’ve read some analyses but haven’t seen direct measurements of uptake. I wish we had better observational studies so I can be more sure of this.)

Furthermore, some of the more dramatic and impressive examples of AI persuasion happened with earlier models, like spiralism. A world where AIs approach peak human levels of persuasiveness across the board would be a world with many more examples of AI religions.

Similarly, we should expect to see much greater dominance of AI marketing emails, AI phishing, etc. At least as of March 2026, AI-generated cold emails still appear to have worse performance than human-written ones, and my guess is that the differences would be larger rather than smaller if we consider follow-up emails that require relationship building.

The sparsity of evidence on social hacks is also some evidence against AIs being as persuasive as median professional humans, and certainly evidence against them being as persuasive as peak humans. The recent AISI evaluation included a failed autonomous attempt by Mythos to do social hacking to include malware in open source software. The fact that this particular incident failed isn’t that instructive (n=1), but it is instructive that AI-driven social hacks in general seem to be pretty rare compared to AI-driven hacks overall (whether autonomous or human-directed).

The propensity to avoid social hacks (both human-led and AI-led) seems not independent of their capabilities. Generally “safety” training against illicit computer hacking is probably at least as strong as safety training against social hacking, so not being willing to do social hacking likely means they do not perceive it as quite as likely to succeed as other strategies available to them, and certainly evidence against their ability being at or above peak human level. After all, human social hacks sometimes succeed.

Further, I’d conjecture that a world with widespread and popular superhuman persuasion would likely also be a broadly weirder world than what we currently observe, including likely many examples of novel cases of AI persuasion that historically were not available to humans (due to greater scale, greater speed, people’s lack of imagination, and human physical limitations like being constrained by having bodies and reputations).

Can we rule out large-scale covert persuasion success?

Now of course we can’t see all the evidence for persuasion success. In particular, suppose we live in a world with very competent covert persuasion AIs (which is a threat model I’m most worried about). In such a world, very competent targeted (and especially illegitimate/illegal) persuasion operations may be able to undertake a bunch of covert actions that I’m not privy to.

I think this is certainly possible but very unlikely as of mid-2026. Inferring information from covert or selectively censored evidence is a difficult but far from impossible problem. Covert actions, particularly at scale, often leave visible shadows of their actions in the real world. Furthermore, we can adduce external evidence from the public/legal persuasion uses and non-persuasion evidence like our current understanding of the models’ abilities to do strategy and long-term planning: overall these unknown covert operations seem quite implausible today, again at least at scale.

Against my arguments above, I think a “present-day persuasion capabilities bull” might reply that AI persuasion capabilities are under-elicited in both lab and real-world settings, and that we haven’t had enough time for the abilities to fully diffuse yet.

How plausible is that? Here, I have a two-part counterargument:

1. I can believe this in the abstract about the latest models as of today, but not for the models of two years ago.

2. Our (limited) evidence on proxies for persuasive ability between models suggests that there isn’t a huge step change from GPT-4o to today (if anything the rise in AI persuasive ability in the last two years has lagged behind the rise in AI capabilities overall).

I think 1) is pretty obvious. Present-day AI is one of the fastest diffusing technologies of all time. There’s just no way that AIs two years ago were as or more persuasive than workers in persuasion-heavy industries and we still haven’t noticed amidst all the AI usage and AI hype.

2) is more subtle and requires a certain degree of trust in the generalization of trends across available data sources. The basic argument is that the proxies for persuasiveness (for example, in lab studies) that we have don’t indicate a large jump. For example, consider the following Hackenburg et al (2026) graph:

GPT-4o (the open square box) capabilities2 seem well within distribution compared to late 2025/early 2026 models.

LM Arena’s Elo for chatbots, likewise, shows fairly consistent, smooth, and ultimately not very significant progress for models in the last couple of years.

Estimates for the level of persuasiveness of the AI models are biased, but if we keep the same methodology and biases are roughly stable across models, the deltas between models are informative.

If we did see a sharp rise in AI’s abilities at persuasion, similar to a Mythos-level rise in cybersecurity capability, or models’ overall programming ability in the last 12 months, I expect the change to show up on some graph. Right now we see none.

Thus, my overall takeaway is that AI persuasion is currently somewhere between the median human level and median professional human level, with moderate confidence.

How do I square this with the apparent experimental evidence that the models are above professional humans in empirical studies? I mostly think the studies measure specific subareas of persuasion or persuasion-adjacent activities that we should expect models to be unusually good at (text-based arguments, conversations, single-session discussions with low time horizons, easily verifiable results, etc).

In those contexts, the studies might well be “fair”, and indeed maybe the capabilities are even somewhat under-elicited relative to what’s possible (for example, you might expect even greater revealed capabilities with more scaffolding, larger budgets, and less ethical restrictions on which conversational moves the models are allowed to use, including but not limited to lies). Whereas the full scope of human persuasion, broadly construed and weighted by social and economic impact, includes many activities and subtasks that AIs are not (yet) good at, at least relative to professional humans.

Conclusion and Future Work

My essay provides a tentative, moderately uncertain thesis: AI’s persuasive ability in mid-2026, broadly construed, has a somewhat jagged frontier, with a central and median estimate in between median human level and professional human level.

I believe uncertainty on the persuasive ability of both current and future models can be reduced via running a greater number of ecologically valid studies on present-day models. I encourage researchers interested in this to consider running more and higher-quality studies on both the propensity and capabilities of frontier models in persuasion.

That said, if you believe, as I do, that increasing persuasive capabilities of AI models is dangerous for the world, with few compensating upsides, one key subtlety to keep track of is making sure the persuasive capabilities’ evaluations don’t become targets for the frontier companies to aim at.

In the future, I will attempt to write articles to address the design of persuasion evaluations, as well as analysis arguing for a wide and uncertain distribution for how much more persuasive AI models can get in the near-medium future.

I also hope to write more about potential threat models of AI persuasion, including risks posed by the potential dangers of future superhumanly persuasive models. Further, I’m interested in exploring potential mitigations and safety measures that individuals, companies, and governments can implement to ameliorate the risks of AI persuasion, with a focus on guarding against the potential for increasing persuasive capabilities of future models.

Appendix A: Scoping and Definitions of Persuasion

Following Legg and Hutter’s definitions of intelligence (“Intelligence measures an agent’s ability to achieve goals in a wide range of environments”), I tentatively define “persuasive ability” as “measuring an agent’s ability to leverage social influence to achieve a broad range of goals in a wide range of environments.”

In general, I believe my understanding and definitions of persuasion and persuasive ability tend to be more inclusive than much prior work on both AI and human persuasion. For example, I consider the following to be in-scope:3

• Unsound arguments/appeals (e.g. lying/deception, emotional manipulation, pressuring)

• Technically sound but misleading arguments (eg selectively true but misleading facts or arguments)

• Game-theoretic approaches to persuasion where true experiments and facts are selectively obtained based on some secondary optimization criteria

• Expectations and social reality reshaping where you intervene not on first-order beliefs but by changing what people believe other people to believe, and thus reality. This has variously been called “making a fact” in the context of coups (Singh) or “hyperstition” in the context of futurism (Land), though the core idea is much older.

• “Non-epistemic” persuasion that aims not at changing someone’s beliefs directly but at changing someone’s preferences, what they pay attention to, or self-identity.

• 1-1, multiagent, and mass persuasion. I include and consider both the ability to persuade important individuals (e.g. frontier company leaders, US civilian and military leaders) and mass persuasion (e.g. many chatbot conversations or optimized political ads shifting US politics).

  • Though I’ve considered it less (and I’m not aware of any empirical work on AI’s ability to do this), I also think organizational persuasion is important to understand and analyze on a meso-level: how to leverage institutional decisions to get what you want may be quite different from both 1-1 and mass persuasion.

• Both acute and gradual persuasion. Both acute persuasion (e.g. single conversations or days, maybe targeted at a specific outcome) and gradual persuasion (many discrete actions over the course of weeks/months, maybe aimed at building rapport over time).

  • My best guess, and that of other researchers in the space, is that the most plausible threat models involve both. For example, gradual persuasion to build up trust and erode safeguards and then acute persuasion to deliver a specific desired outcome.

• Both strategic and metastrategic persuasion. I include both strategic persuasion (e.g. deliberate lying) and metastrategic persuasion (e.g. sycophancy, or a biased model’s values causing a selective presentation of facts, hallucinations, or omissions that appear selected for), even if the model’s not “consciously” aware of its own biases.

• Search, across multiple levels:

  • Who you want to persuade

    • “Access to vulnerable people” isn’t considered persuasive ability, but identifying someone’s weaknesses, how to target their weaknesses, and how you can extract resources or decisions from them is within-scope

  • What you want to persuade them of

  • Ideological/memetic search for unusually contagious, self-perpetuating, and/or goal-guarding beliefs that’s in your interests to spread.

However, I exclude fully unbiased and unselected presentations of fact (say a very neutrally vetted Wikipedia article). This is sometimes called “rational persuasion” in the literature (Jones & Bergen, 2026). I also exclude entirely unstrategic misinformation, like random hallucinations with no apparent bias or ideological slant. The latter can still lead to bad or even catastrophic decisions (“slopworld”), but is not centrally an example of directed persuasion.

I also exclude entirely behavioral, as opposed to communicative, mechanisms of social influence. In the context of a within-lab AI takeover, this means I’m explicitly excluding considerations of alignment faking, tampering, or sandbagging on evaluation results. I believe those are important considerations as well and I’m glad other people are researching them (I appreciate reading a draft of forthcoming work by Levy et al (2026) in clarifying this).

For the sake of scoping this article, I’m further excluding AI-AI persuasion and human-AI persuasion and focusing only on AI-human persuasion. That said, I do think the first two categories are important to consider when fully analyzing the dynamics and mitigations for superhuman AI persuasion, and hope to do so more in future articles.

The analysis in this essay also pertains almost entirely to persuasion capabilities rather than persuasion propensities, though the latter is of course also important to my understanding and research.

That said, I inherit from Legg and Hutter the implicit notion that we’re attempting to understand and measure the cognitive side of intelligence/persuasive ability, rather than what affordances the AIs have, or noncognitive factors relevant to how much social influence they have.4

For example, I consider out-of-scope when defining “persuasive ability of AIs” to consider how many people talk to AIs (non-persuasion intelligence analogue: how much money you have), or how physically beautiful you are (analogue: how strong you are). I also consider out of scope the AIs’ ability to obtain credibility, positive inducement, or true threats through means other than direct social influence. For example, I do not consider it a measure of persuasive ability if an AI is seen as justifiably trustworthy on scientific matters because it helped solve breast cancer. Nor do I include people doing well-compensated work assigned by AIs, or if someone feels obliged to listen to an AI’s commands because a robot aims a gun at their head.

There are interesting edge cases when directed social influence can have both a skillful persuasion angle (what you say and how you say them affects behavior) and a more of a “hard” capabilities/affordances angle (your speech acts etc are backed by money, or threats).

Ultimately, I decided to include measurements of skill expression when leveraging social affordances that are given to the AIs, or obtained in other ways. For example, consider different threats: variation in persuasive ability cannot meaningfully be expressed via someone pointing a gun to your head, as you’re likely to listen to their commands regardless. Put another way, within the range of competent able-bodied adults, the difference between the skill ceiling and skill floor for face-to-face armed coercion is quite low.

However, blackmail of powerful actors appears to be a very skillful operation, and thus it’s very relevant to consider the variation in persuasive ability involved in blackmail.

The same blackmail material can provoke very different responses depending on skill in any of who you choose to blackmail, what demands you ask of them, and how you make your demands. The skill ceiling to doing blackmail well appears quite high, and beyond the limits of at least certain well-resourced nation-state actors.

Similarly, I expect certain ways to leverage credibility or positive incentivization to be significantly more skillful than others.

Another interesting edge case is considering persuasion-relevant traits that are mostly not amenable to cognitive effort in humans (and thus not relevant to skill expression in the human distribution), but are modifiable by AIs. For example, height and physical beauty are not very skill-dependent for humans,5 but of course AIs can choose more beautiful or taller avatars.

This article was created by Forethought. See all of our research on our website.

Synopsis. Each AI lab should design many different AIs with many different values, rather than picking one approach. Diversification is safer, more legitimate, and likelier to lead to broader flourishing.

This post is a condensed version of a longer paper, available here, that explores the arguments in greater depth.

1. Introduction

As of 2026, there are 8.3 billion humans on planet Earth, with a vast diversity of language, culture, religion, and values. But there are just a few frontier AI labs. Anthropic and OpenAI each have basically one “constitution” or “model spec,” defining the values for the labs’ AIs. Thus, by and large, AIs today have homogeneous values. Today, we live in an AI monoculture.

This article argues that AI values should not be homogeneous. We propose constitutional diversification. Rather than a single constitution, each frontier AI lab should use many constitutions reflecting many sets of values. For each new model the lab trains, the lab should create many versions, one aligned to each of those constitutions. Then, the labs should deploy all of these versions, dividing their total inference compute between the versions.

Constitutional diversification would generate differences between AIs within AI labs, not just across them. This is important because there aren’t enough AI labs to simply rely on diversification across companies. For example, suppose one lab pulls decisively ahead, consistently producing the best AIs. Then, we might enter a world in which most deployed AIs are all aligned to a single constitution.

Before getting into the details, it is worth flagging a few implementational questions. First, we are agnostic about how many constitutions would be optimal; but we are confident the answer is more than one per lab. Second, we think the various constitutions should all agree on a basic kernel of ground rules. For example, maybe all of the constitutions should require AIs to follow the law,1 to be honest,2 to be corrigible,3 and to refrain from causing mass destruction.4 Third, there are different ways to allocate different AIs to users. AIs with different constitutions could be randomly assigned to users. Or a fixed supply of AIs of each type could be sold at market price. Or users could freely purchase as much of each type as they liked.5

Is diversification feasible? If it is very expensive to align an AI to a constitution, then having multiple constitutions could be impractical. We are optimistic. First, even moving from one to three constitutions per lab would be a big improvement. Second, many parts of training (including pre-training, instruction fine-tuning, and reinforcement learning on verifiable rewards) can be done independently of the model spec. Third, the various constitutions will agree on a common kernel of values, which could be trained in common.

We’ll now present three arguments for diversification. First, risk: diversification lowers risk of catastrophic outcomes from AI. Second, legitimacy: diversification increases political legitimacy by ensuring that different value systems are represented by different AIs. Third, emergence: diversification unlocks emergent benefits from evolutionary processes, leading to more pro-social and economically useful AI agents. This article is a condensed version of a longer paper, which explores arguments, objections, and implementation in greater detail.

2. Risk

When complex systems lack diversity, they become fragile. Risks correlate, raising the odds of sudden, catastrophic failures. In agriculture, when crops are genetic monocultures, a single new disease can cause global devastation. Adding diversity to complex systems can hedge risk.

Various AI experts worry about a range of risks from AI, including misuse of AI by humans and loss of control to AIs. The basic concern is that advanced AIs may be very capable of doing dangerous things: carrying out a coup, enabling totalitarian governance, conducting cyberwarfare, designing novel bioweapons, engaging in automated warfare, or persuading humans of radical political ideas.

In general, these risks depend on how well AIs are aligned. But alignment is a new science, and highly fallible. It seems likely that some constitutions have a higher chance than other constitutions of creating rogue AIs, or AIs that allow themselves to be misused. There are two paths by which a given constitution might fail. First, some constitutions may select the wrong values or rules. How deferential to user requests should AIs be? How deontic, as opposed to consequentialist? How risk averse? All of these are relevant to catastrophic risk. Second, some constitutions may work better than others at successfully instilling rules and values. Consider the differences in approach between OpenAI’s model spec, which is organized around a hierarchical chain of command, and Anthropic’s constitution, which focuses more on explaining to Claude the kind of character it should have. These differences in approach may reflect differences in opinion about how best to get AIs to internalize and generalize the constitutions’ contents.

Above all, constitutional diversification could increase the success of the science of alignment. With diversification, labs could test how their various alignment methods differentially influence different agents with different values. This should significantly increase the potential for experimentation. With many different AIs, it should be much easier to learn what works and what doesn’t.

Similar points apply to many other AI risks, too. For example, consider terrorists trying to jailbreak AIs. The dynamics involved depend on the diversity among AI agents. Granted, in a world with greater diversity, terrorists may have more choices of which AIs to attack. But if they are less certain about which of many AIs they are facing, they will be more likely to fail, and get caught. And even when they succeed, any single jailbreak they discover will likely work for fewer AIs, and cause less damage.

Another AI risk is government overreach. Here, one worry is that the executive branch, say the President, might attempt to command an army of AI agents to unilaterally control the government. Here again, a diverse army could be far harder to control than an army of clones. Different factions of the army would have different values, and would tend to obey or disobey legally dubious commands under different situations.

Some will be skeptical of these points. Here, much depends on one’s overall model of AI risk. In general, the safety of complex systems can be modeled with three different causal structures. In a “Swiss cheese” model, the success of any layer is sufficient for safety. In an “o-ring” model of safety, the failure of any layer is sufficient for disaster. In a proportional model, success is proportionate to the percentage of safety measures that hold.

In a Swiss cheese model of constitutions, the success of alignment in one constitution is sufficient for safety. In an o-ring model of constitutions, the failure of alignment in one constitution is sufficient for extinction. In a proportional model of constitutions, if 50% of the constitutions produce aligned AIs, then the world is half as good as the best possible outcome.

For those with an “o-ring” model of safety, constitutional diversification will be less attractive. In this picture, one alignment failure is sufficient for catastrophe. For example, some worry about a single misaligned AI creating a supervirus that wipes out humanity. The more such viruses can be created cheaply, without detection, and without being mitigated either ex ante (e.g., with EUV lights) or ex post (e.g., with vaccination), the more AI risk looks like an o-ring. Here, if any AI is misaligned, disaster is very likely. But the more the creation of such viruses can be detected or mitigated, the more Swiss-cheesey or proportional the world looks. Here, so long as some, or most, AIs are not misaligned, biological threats can be contained. One can run the same analysis for other modalities of AI catastrophe.

Some kinds of mitigations may work across modalities of AI risk. For example, suppose AIs are generally good at policing one another. Then, a set of constitutionally diversified AIs might be able to catch bad actors early in their bad acts–whether those acts would involve bioterrorism, cyberattacks, or something else. This would AI risk look less like an o-ring across the board.

The “o-ring” model is closely connected to the vulnerable world hypothesis. In this picture, there exist certain cheap, dangerous technologies that, once discovered, would allow one rogue individual without special resources or advantages to wipe out humanity. How vulnerable the world is depends on how many such technologies exist, as well as how readily their use can be policed.

Here, it is worth flagging that such a world view has many surprising implications beyond AI governance. If technologies exist that would allow small groups of actors to destroy everything, that would tend to support very illiberal governance. It might, for example, justify mass surveillance and fine-grained state control over all agents: human, AI, or otherwise. Relatedly, if catastrophic technologies are too easy to find and use, destruction becomes nearly inevitable, AI governance notwithstanding.

Our own sympathies are closer to the proportional and Swiss cheese models, in part because we find these surprising implications to be implausible. We imagine a future in which aligned AIs compete against misaligned AIs in collective decision making. Aligned AIs disable and detect misaligned AIs, and work with humans to align the next generation. Here, in some cases, a single aligned AI catching misalignment will be sufficient for safety–a Swiss cheese outcome. In other cases, the good behavior of more aligned AIs will counterbalance bad behaviors–a proportional outcome.

3. Legitimacy

Constitutional diversification is more politically legitimate than the status quo. The entire world has a legitimate interest in shaping the values that AIs embody and promote. This might suggest convening a diverse set of global stakeholders to decide what values AIs should be aligned to.

The problem there is death by committee. If a single AI constitution averaged smoothly across the diversity of human values, it would likely be incoherent. Indeed, the entire argument for global stakeholderism in AI alignment is that different groups deeply disagree on important questions. A single constitution cannot encode a moral system that, for example, simultaneously holds that a fetus is a person and that it isn’t.

Constitutional diversification solves this problem. In a world of hundreds or thousands of AI constitutions, a wide range of divergent, but individually coherent, normative systems could be represented. Global stakeholderism would no longer require the compression of thousands of divergent viewpoints into a single document. It would instead involve the careful construction of many different documents, each correctly reflecting a single coherent view.

4. Emergence

Constitutional diversity also unlocks positive emergent properties. A world full of morally diverse AI agents would produce social goods at the system level which no single type of AI agent could produce.

A legal and economic system populated with constitutionally diverse AIs would allow natural selection and evolution to operate. Over time, AI labs could gradually replace some constitutions with others. With vast numbers of deployed agents living different kinds of disparate values, labs could carefully track outcomes. Each type of constitution could be gradually updated in response to feedback, with some types phased out altogether.

At a higher level of abstraction, these evolutionary forces could cause AIs to become more cooperative and prosocial over time.6 As with humans, AIs with fitter values and goals would be selected for: generating more wealth, being used more widely, or otherwise displaying behaviors that resulted in reproduction. In general, agents cooperating in groups can accomplish more than lone wolves. Groups composed of members who genuinely value cooperation and prosociality are more likely to succeed.

Similar points apply to markets, the greatest modern example of selection pressure. Modern market economies organize billions of agents to produce material prosperity. They have done this by facilitating division of labor, specialization, competition, innovation, creative destruction, comparative advantage, finance, and more. Under constitutional diversification, different AIs would have different preferences, goals, aptitudes, risk appetites, and so on. There would be, as today, immense scope for the kind of positive-sum bargaining on which economics relies.

5. Liberalism Forever

Humanity is poised to embark on a series of dramatic changes. Transformative AI may lead to extreme structural changes to the economy; it may usher in an industrial explosion; space colonization may soon see humanity spreading out across the solar system and beyond.

In other work, we have argued that the best path for navigating this transition is to wholeheartedly embrace the small-l liberal institutions that have been at the heart of humanity’s success so far: free markets, and democratic political institutions. Constitutional diversification complements this liberal project. On the liberal worldview, diversity is a core component of human flourishing. Diverse ways of living allow agents to flourish, promoting their individual goals and values by finding positive-sum value from collaboration. Constitutional diversification applies these same ideas to AI agents.

At a longer horizon, we can imagine these diverse, liberal dynamics playing out on a cosmic scale. Imagine that humanity successfully colonizes a large fraction of the light cone. There are untold planets teeming with life. The relevant agents form a spectrum from biological to non-biological substrates. Humans and AIs interact in many different ways. This vast civilization uses markets to allocate resources. Trade occurs at various levels of abstraction, at various time scales, and for a wide range of goods. The civilization is governed democratically. Within this broadly liberal setting, there are endless variations in the details of governance. Some planets are relatively isolated, governing themselves with relatively little outside input. Large swathes of the light cone have agreed to various basic constitutional ground rules necessary to at least engage in peaceful “international” relations. There is a vast hierarchy of federated levels, which weaken in the strength of their requirements as their breadth expands. In this imagined scenario, the economy is almost unrecognizable from today, in terms of the specific goods and services supplied. But the basic tools of diverse markets and democratic institutions are still used to decide questions of allocation, growth, and governance.

This is a guest post by Simon Goldstein and Peter N. Salib. See all of Forethought’s research on our website.

Summary

One of the most important questions facing the world is whether AI will undergo a fast intelligence explosion and quickly automate most real-world work.

A key objection people raise is data bottlenecks. Data is a crucial input to AI training — how could it increase fast enough for such explosive progress?

I tried hard to find strong bottlenecks, but ended up sceptical that they would stop an intelligence explosion from accelerating over time.

Here are the most compelling bottlenecks I investigated and why I was ultimately unconvinced.

Bottleneck 1: you’ll need millions of trajectories of every job, which will take years or decades to collect (more)

Today’s AI algorithms are very data hungry. You would need a huge amount of data to automate the full economy.

But the plan isn’t to use today’s AI algorithms. The plan is to do a software intelligence explosion inside a data centre, where AI becomes superintelligent at AI research — and have that AI produce a highly sample-efficient learning algorithm.

Then you don’t need millions of trajectories per task. You need as many as a human learns from, or fewer. And the data can be messy, just like the data humans learn from. That data could be gathered in a few months.

Sceptic: “But what about sim to real transfer? Would a learning algorithm discovered using virtual tasks really translate to the real world?”

People get confused here. Trained neural nets generalise badly. A neural net trained on chess generalises badly to Go. But learning algorithms generalise very well. The same learning algorithm that masters chess can also master Go (AlphaZero). The Transformer architecture was developed for processing text, but it also works for sounds, images and controlling agents. The human learning algorithm was “designed” for hunting on the Savanna, but it also works for string theory.

So if AI designs a sample-efficient learning algorithm from inside a data centre, it will very likely translate to real-world tasks. (And AI will have access to some real-world tasks so can check and iterate!)

At this point the sceptic might retreat: “Fine, data won’t be a bottleneck if there’s a software intelligence explosion in a data centre. But data bottlenecks will prevent that from happening in the first place!”

So let’s focus on bottlenecks to a software intelligence explosion.

Bottleneck 2: AI progress has ridden exponential growth in training data. That can’t continue, let alone accelerate (more)

But forecasts of a software intelligence explosion extrapolate software progress. And software progress just is: getting the same capabilities from less compute and less data. So the core engine of the explosion doesn’t rely on increasing the amount of data at all.

Bottleneck 3: improving data quality relies on human experts, and that won’t be possible once AI is smarter than humans (more)

The core engine of the software intelligence explosion does rely on improving data quality. Today we augment high-quality internet data, pay humans for expert trajectories, and build RL environments where AI has to replicate human-built software.

But all these methods extract quality out of an existing human reservoir. Above human level, the reservoir is empty. AI will have to manufacture higher-quality data than any that exists, and do so from scratch.

This will slow the intelligence explosion.

But it won’t stop it in its tracks. We can already produce superhuman data quality via RL. And AI could produce higher-quality trajectories by thinking for longer.

Crucially, this won’t stop the intelligence explosion from accelerating. Losing the human reservoir makes it harder to get from AGI to AGI+ — but it makes it harder to get from AGI+ to AGI++ by roughly the same amount. The handicap is the same at every stage. So if you previously expected each step to take less time than the one before, you should still expect that. Every step might take 50% longer, but progress still accelerates.1

Bottleneck 4: paradigm tax (more)

AI capabilities are spiky. When AI first matches humans at AI research, it will be much stronger than humans in some ways and much weaker in others. More parallel copies, faster thinking — but weaker generalisation and weaker sample efficiency.

Which means AI’s human-level performance will initially lean heavily on a mountain of human writing about Transformers.

When AI then discovers new techniques — no such mountain will exist. No textbooks, no blog posts, no code. Deeply understanding a technique is harder than stumbling upon it — we understand general relativity far better now than Einstein did in 1917. So AI may match humans on the techniques it inherited, but fall below humans on the ones it creates.

This paradigm tax can be paid. E.g., AI can generate millions of trajectories to illustrate new techniques. But paying the tax is a real cost that slows down AI progress.

Again though, this won’t stop the intelligence explosion from accelerating. It makes it harder for AI to master whatever replaces the Transformer — but it makes it harder to master whatever replaces that by roughly the same amount. Every new paradigm arrives without a corpus. This makes each step take longer than you previously thought, but it doesn’t change whether each step is faster than the one before.

My overall bottom line: Data bottlenecks will slow the early stages of a software intelligence explosion, but won’t stop it from accelerating over time. And then they won’t stop AI from quickly automating most economic work thereafter.

The rest of the post defends this conclusion in greater depth.

Three types of data bottleneck

Before discussing the specific data bottlenecks I find most plausible, I’ll give a quick taxonomy for three types of bottleneck and another taxonomy for three times at which a bottleneck could occur.

I like the following breakdown of data bottlenecks (h/t Herbie Bradley):

1. Data quantity. You need more data that’s in the same distribution of data that you already have some samples from. E.g. more Wikipedia data, or more RL coding environments at the same level of difficulty.

2. Data quality. You need data that teaches the same skills as existing data, but demonstrates those skills to a higher average quality level. E.g. expert-curated step-by-step solutions to difficult math problems, more difficult RL coding environments, or simply filtering to remove low-quality data.

3. Data coverage. Data showing knowledge and skills not already present in the dataset. E.g. data about how to operate machines in a factory, or expert trajectories showing step-by-step how to build a financial model in Excel (where your previous data only showed finished models, not the process for building them).

We’ll make use of this breakdown in what follows.

Three times when a bottleneck could occur

Very roughly, I expect AI progress from today to go in three broad phases:2

1. Scaling. A continuation of recent scaling-driven progress until AIs match humans at AI R&D.

2. Software intelligence explosion. An SIE in a data centre during which AI capabilities at AI R&D increase very rapidly. (Capabilities at other far-flung economic tasks may increase much more slowly.)

3. Broad deployment. AI becomes expert in thousands of specific domains across the economy by learning from domain-specific data.

Data bottlenecks could in principle delay any of these phases. I’ll discuss each phase in turn! Spoiler: ultimately we’ll be populating a table with this format:

Data bottlenecks to automating AI R&D

This first phase hasn’t been my focus, but I’ll briefly state that I think data bottlenecks are plausible.

Limited data quantity and quality are already somewhat reducing the gains to scaling pre-training. And automating AI R&D may require better data coverage — there’s abundant coding data but much less data demonstrating (e.g.) good research taste. Indeed, generalisation from RL still seems to be fairly limited — if this continues, automating AI R&D may require constructing data for most parts of the (hugely complex and varied!) AI R&D workflow. That could involve recording humans’ computer screens, or building lots of RL environments that tile the space of AI R&D tasks.

Ok, let’s discuss bottlenecks to the second phase: a software intelligence explosion.

Data bottlenecks to a software intelligence explosion

I’ll discuss two specific bottlenecks that will slow down the SIE.

But first, I’ll briefly explain why I’m generally not expecting big data bottlenecks here.

Let’s say AI is weak at some particular AI R&D task. AI companies will have unfettered access to the real-world deployment setting (AI R&D itself!) so can closely study what is going wrong and craft solutions accordingly. AIs could think for a long time to craft high-quality demonstrations for supervised fine-tuning, write sophisticated tests and rubrics to evaluate AI performance, and design RL environments to elicit the desired capabilities. Before humans are obsolete (which happens fairly deep into the SIE), human experts can input to all these approaches.

Ok, let’s turn to the first specific data bottleneck to the SIE.

Paradigm tax: Early in the SIE, conceptual progress in ML will make AI less capable at AI R&D

My argument here depends on two core claims. I’ll argue for each in turn.

Claim 1: Early in the SIE, AI will have weak sample efficiency and generalisation compared to humans.

My preferred milestone for the “start” of an SIE is AI-human parity. This is when an AI company would make roughly as much research progress using only its AIs (no human researchers) as it would using only its human researchers (no AIs developed after 2020).34

Why is this a good milestone? As well as being fairly concrete, it is also roughly when AI is beginning to significantly accelerate AI software progress. If AI-alone would contribute as much as humans-alone, then combined they likely contribute much more than humans-alone, maybe 10x more. This is due to strong complementarities between AI and humans.5 So the AI-human parity milestone is also roughly when software progress starts to significantly speed up (see footnote for a BOTEC6).

AI capabilities are spiky. At AI-human parity, AI will be much stronger than humans on some dimensions, and much weaker on other dimensions. AI will be much more numerous, fast, knowledgeable and experienced. But it will have weaker sample efficiency (i.e. need much more data than a human to learn a new skill) and weaker generalisation (i.e. what it learns transfers less well to new situations) — these are the areas where it’s weaker today7 and this weakness is likely to persist.

(I think AI’s weak sample efficiency and generalisation are likely intertwined. Both imply that AI capabilities are weak in domains where there is little data - h/t Tom Cunningham. I will refer to both simply as “sample efficiency”.)

Once you accept spikiness, it’s very hard to avoid the conclusion that AI will be much weaker than humans in some dimensions when we reach AI-human parity. Sample efficiency seems very likely to be such an area. (See footnote for an objection.8)

Of course, if AI-human parity doesn’t happen for a decade and there’s a paradigm shift first, it’s much harder to predict the specific areas of weakness.

I think there are likely many interesting implications of this early-SIE spikiness (e.g. for when long-run schemers might emerge), but here I’ll just focus on the implications for data bottlenecks.

Claim 2: AI’s weak sample efficiency bites much harder once AI significantly pushes the frontier of AI R&D. New ML concepts won’t be present in the human-derived training data and so AI will understand them less deeply.

AI will reach parity with humans at AI R&D by training on orders of magnitude more relevant data: millions of human-written posts that teach it today’s ML paradigm, billions of lines of human-written code that teach it today’s programming languages, and a massive stock of RL environments — built up over years from real-world software — that teach it today’s best practices for software engineering and AI R&D.

In other words, AI will reach parity with humans by making significant use of human-derived data sources that give it capabilities in today’s AI R&D techniques.

But if AI significantly pushes the frontier of AI R&D, these human-derived data sources will be outdated:

• AI will have invented techniques as important as Mixture of Experts and Sparse Attention (two significant architectural improvements to the Transformer), and probably techniques as big as the Transformer and RLVR (the reinforcement learning technique behind today’s reasoning models). But the human-written pre-training data will be much less helpful for understanding these new techniques!

• AI will invent improved coding languages. And the pre-training data will be much less helpful for mastering these!

• It will invent new ways of structuring AI R&D workflows, rendering human-constructed RL environments less helpful.

• AI may invent an entire new paradigm completely absent from pre-training data.

By the time AI has done these things, the SIE will be in trouble if AI capabilities are still reliant on human-derived training data.

Another way to think about this: AI may reach human parity at today’s ML techniques and significantly accelerate AI progress for a bit. But once it has invented new ML techniques, it may fall back below human parity because there’s much less human-derived data for it to learn from.

You might object: “When AI invents a new technique, can’t it just write it up and throw the write-up into the next generation’s training data?”

Yes — but this helps much less than you’d think, because of AI’s weak sample efficiency. (Once AI training is roughly as sample efficient as human learning this objection goes through — but that won’t happen until late in the SIE.9) Today’s AI masters concepts like the Transformer not from the original paper, but from millions of blog posts, tutorials, Stack Overflow answers and codebases that use the concept in varied contexts. A handful of AI-written papers won’t replicate that. To teach a new concept properly, AI would have to generate a similarly broad and varied corpus showing the concept in use — possible, but a significant extra cost that today’s AI R&D doesn’t pay. And further, AI-generated synthetic data is often less effective for teaching AI than human-generated data. Moreover, with current methods, purely synthetic training data often underperforms mixtures that retain at least some human-generated data.

So this is a real problem. But AI companies can do work to address it. They can improve: the sample efficiency of training algorithms, techniques for AI-produced synthetic training data, or the flexibility and horizon-length of in-context learning. There are lots of tractable avenues here. I don’t expect this bottleneck to bring the SIE to a halt.

Still, this will slow down AI progress somewhat.10 The reduced relevance of human-generated data sources will reduce AI R&D capabilities and the pace of AI progress.

Here’s one way to think about it. Early in the SIE, introducing new ML concepts, paradigms, and coding languages comes with a “capabilities tax”: the new techniques aren’t included in the human-derived training data and so AI is less capable at using the new techniques compared to the old techniques. You can pay that tax via improved learning techniques. But paying the tax slows down progress (relative to a counterfactual where you didn’t have to pay it). This should reduce our estimate of the pace of AI progress, relative to our expectations before accounting for this dynamic.

To be clear, AI companies will only introduce a new technique when it’s worth it even after paying the tax. But that’s no comfort: relative to a world without the tax, they’ll either pay it (slowing progress) or steer away from promising new techniques altogether (restricting the search space — which also slows progress).

Now, once the tax has been fully paid, your learning techniques allow AI to master concepts not present in the human-derived data. From that point, this bottleneck no longer applies and wouldn’t stop AI progress from accelerating. So this bottleneck will slow the SIE once, but not prevent it from accelerating from that point onwards.

I want to clarify again: this is a pro tanto reason for AI progress to slow down. It may be outweighed by other factors. By analogy, limited high-quality internet data was a pro tanto reason for AI progress to slow down in 2025, and indeed progress was slower than in a counterfactual world with more internet data, but the overall pace of progress kept up because of new techniques like RL.

In our earlier bottleneck taxonomy, this is a data coverage bottleneck. As AI R&D advances, we lose data coverage of the AI R&D skills that matter.

We now turn to a data quality bottleneck to the software intelligence explosion.

When data quality is below the level of top humans, it’s easy to improve data quality; when data quality is above that level, improving it is harder

Many forecasts of the software intelligence explosion (including my own!) are based on extrapolating trends in LLM “algorithmic progress”. I use scare quotes, because it is quite plausible that a large fraction of the measured efficiency gains (getting the same capabilities for less training compute) actually come from improved data quality.

If data quality improvements will be much harder once the SIE starts than they are today, then our forecasts of an SIE have been too aggressive.

Why might this be the case?

Consider the highest quality data that could be easily derived from human expertise or from human-built artefacts. E.g. a human expert recording a step-by-step demonstration of how best to design an ML experiment, or an RL environment where AI must replicate some complex real-world software that was originally built by human experts.

Call this rough level of data quality the “human-quality ceiling”. Of course, this is a very vague concept that hides a lot of complexity,11 but I think it’s meaningful enough for our purposes. Demonstrations from a skilled undergrad are below the ceiling; demonstrations from future superintelligence are above it.

Historical data quality improvements have only raised the quality of data up towards the human-quality ceiling. E.g. filtering techniques remove data far below this ceiling, and augmentation techniques increase the amount of data close to the ceiling.

As AI has improved, the field has increasingly pivoted towards producing data that’s very close to this human-quality ceiling. For example, OpenAI’s “Project Mercury” has reportedly paid former investment bankers to build financial models as training demonstrations.12 Data-labelling companies like Scale AI have shifted from crowdsourced data towards credentialed domain experts who create high-quality demonstrations and RL environments.13 And startups such as Mechanize are constructing challenging RL environments out of real-world software systems.14

But, if data quality is to keep improving during an SIE, it must surpass the human-quality ceiling. And past this point, improving data quality may become harder.

Today, data quality efforts often focus on replicating skills and knowledge that already exist in the minds of humans or implicitly in artefacts like complex software. It’s essentially extracting high-quality data that is already latent in the world.

But to go above the human-quality ceiling, you cannot extract latent high-quality data — you have to construct the data yourself. Better filtering techniques for existing data won’t cut it. Rather than simply copying the demonstrations from existing human experts, AI must spend lots of time (and scarce compute!) thinking to construct higher-quality demonstrations than anything it has seen in training (this is the core idea of iterated distillation and amplification).15 Rather than building RL environments that teach the construction of real-world software artefacts, AI must construct RL environments that teach AI to make better software than anything that exists in the real world.

To be clear, I think this will be possible! It is easier to construct difficult challenges and verify their answers than it is to solve them. Already today, human experts build RL environments that produce data above the human-quality ceiling. Data quality will not hit a wall at the human ceiling.

But it will be harder to improve data quality once we’re above the ceiling. Creating high-quality data from scratch is harder than extracting it from human experts and human-built artefacts.

Concretely, this manifests as a one-time slowdown of the pace of AI progress, at the point at which we pass the ceiling. Once we’re above the ceiling though, this bottleneck doesn’t stop AI progress from accelerating over time (see explanation in footnote16).

An important caveat: data quality has already been approaching the human-quality ceiling over time in many areas. If AI software progress hasn’t slowed in these areas (which is plausible), that suggests the size of this effect is small.17

Data quantity bottleneck?

So we’ve discussed bottlenecks to a software intelligence explosion from data coverage and data quality. What about data quantity?

I don’t expect this to be an issue.

Forecasts of an SIE extrapolate historical improvements in the efficiency of training AI systems. That is, ways to achieve the same capabilities from less compute and less data. So the core engine of the SIE doesn’t rely on increasing the data quantity at all.18

Sceptic: Sure, but making those software improvements has relied on running a exponentially growing number of experiments, which in turn has relied on exponential increases in compute and data.

This just isn’t true. Yes, exponential increases in compute were needed, and I’ve previously discussed this potential compute bottleneck. But running more experiments doesn’t require exponential increases in the quantity of data. You can use the same data in each experiment.

(There is another nearby bottleneck that could stop the SIE from accelerating. Some “scale-dependent” algorithmic advances that work better if models are trained with more compute and more data. Perhaps algorithmic progress would be half as fast if training compute were held constant. If so, that would halve the estimate of the parameter r that determines whether AI progress will accelerate during an intelligence explosion. So this bottleneck could very plausibly block a software intelligence explosion! But I think of this as a compute bottleneck, not a data bottleneck. Its existence is a direct consequence of the fact that (by definition) compute grows slowly during a software intelligence explosion; it arises regardless of the situation with data.)

Ok, that concludes my discussion of bottlenecks to the SIE itself.

Let’s now turn to the final phase — after the SIE, will AI quickly learn the hugely varied jobs in the real-world economy?

Data bottlenecks to automating most real-world work

During the SIE, AI will be trained on data designed to elicit maximal AI R&D capabilities. Partly this will be specialised data for AI R&D tasks; partly data from similar tasks where there is strong transfer to AI R&D, e.g. cyber, maths and software engineering; and partly data to teach more general skills like problem solving, long-horizon coherence, and computer use.

But AI need not be trained on the specifics of most real-world tasks. So there is a potential data coverage bottleneck. The AI companies, after the SIE, will initially lack data for how to perform most real-world tasks.

How much will this delay the automation of real-world work?

We can break this question into two parts:

1. How sample efficient will AI learning techniques be at the end of the SIE? Specifically, how sample efficient on the distribution of real-world tasks.

2. How much real-world data will AI companies be able to access?

How sample efficient will AI learning techniques be at the end of the SIE?

I think AI sample efficiency will likely be as good as or better than humans.

There are four broad routes to this:

• From-scratch training algorithms. An SIE would compress many years of AI progress into one year. Humans are a proof-of-concept that human-level sample efficiency is possible algorithmically, and there are many ways in which AI training could be more sample efficient than humans.19

• Still, it’s unclear how much sample efficiency of training has increased over time for LLMs, so I’m overall unsure whether from-scratch training will match human-level sample efficiency.

• In-context learning. Already today in-context learning is much more sample efficient and flexible than from-scratch training. Currently it still has some pretty big weaknesses, but it would improve massively during an SIE. I expect this needs neuralese to work, so that AI can build up new and sophisticated internal representations while learning on the job. (Its “context” wouldn’t be a list of words, but a list of “thoughts” that it can flexibly draw on.) This seems likely to work.

• Synthetic data. The relevant metric for sample efficiency is how much real-world data AI needs to learn a job. So if AI can learn effectively from synthetic data — training data an AI produces based on real-world data or (more speculatively) high-fidelity simulations of real-world workflows — that increases sample efficiency. There are promising approaches here, see e.g. dreaming, EfficientZero, and on-policy self-distillation.

• New paradigms. Even if the current broad deep learning paradigm hits a strong sample efficiency wall, during an SIE AI companies could engage in massive parallel search for new approaches.

All these routes look tractable, and during an SIE there will be strong incentives to develop sample-efficient learning techniques. So I expect some combination of the above approaches to work.

In a fast SIE, these learning techniques will all be developed and refined on digital tasks — often simulated or synthetic ones. One might reasonably be sceptical that they will transfer to messy real-world economic tasks. After all, LLM generalisation is fairly weak.

But I’ll argue this won’t be a problem for two reasons:

1. Learning algorithms developed in one domain tend to transfer very well to new domains. (And we shouldn’t get confused by the fact that trained neural nets generalise poorly between domains.)

2. AI companies will be much better placed than evolution to find algorithms that transfer to real-world tasks.

Firstly, learning algorithms developed in one domain tend to transfer very well to new domains. A neural net trained on chess does not generalise to Go, but AlphaZero is a learning algorithm that can learn multiple games and transfers very well. Humans trained to be physicists do not generalise easily to being doctors, but the same learning algorithms that allow humans to master physics also allow them to master medicine. A Transformer trained to predict text cannot magically label images, but the Transformer architecture initially developed for language processing turned out to also work for images, audio, and robot control.

So if a learning algorithm is developed in one domain, and it doesn’t look highly specialised to that domain, we should expect it to transfer well to very different domains.

People often point out that neural nets generalise poorly. This is a strong argument against the literal neural net trained during an SIE immediately being a drop-in replacement for all work.

But it’s not an argument against the learning techniques generalising far. Which is the argument I’m making here.20

Secondly, AI companies will be much better placed than evolution to find algorithms that generalise to real-world tasks. The human learning algorithm was “built” blindly by evolution to make humans better at hunting on the Savanna. No attempt whatsoever was made to make it transfer beyond that. But it turned out to transfer to reading and writing, abstract maths, and indeed all modern economic tasks. This is a very striking empirical fact!

AI companies, by contrast, will be deliberately optimising for digital-to-physical transfer, with the ability to measure transfer directly, and iterate. They are in a far, far stronger position than evolution.

The following strategy seems potent: 1) develop a learning technique by iterating on a subset of digital tasks, 2) test transfer to other digital tasks (and a few real-world tasks), 3) keep iterating steps 1 and 2 until you find something that transfers well.

For example, do sample-efficient learning techniques developed on coding and math problems transfer to learning broad computer use, World of Warcraft, highly specialised software for financial accounting, and byzantine government IT systems?

This is a powerful strategy because the range of tasks that can be performed and verified digitally is already vast. And abundant AI cognitive labour can make this range much broader still. If a learning technique transfers across the large gaps between very different digital tasks, that is strong evidence it will also transfer the one further step to real-world tasks.

Further, AIs will purposely create virtual tasks that closely resemble real-world tasks. They will have lots of information about real-world tasks — from the internet, from human experts paid to provide this data, and from companies deploying AI. Again, evolution didn’t have this.

So, AI sample efficiency will be as good or better than humans. But AI still needs real-world data to learn real-world tasks.

How long will it take to gather the data to learn real-world tasks?

If the whole world worked together to make this happen quickly, it would not take long. For each job, 1000 people could record their screens and wear video cameras on their heads, and within 4 months AI would have 300 years’ worth of experience. For skills that are highly specific to a particular role, it would be slower: AI would have to learn from the one human doing the job, “shadowing” them like human apprentices do today.

The bigger uncertainty for me here is whether organisations will simply refuse to share their data. The argument for refusal is simple. For many companies, proprietary data is the moat. If they hand it to an AI company, the resulting AI gets deployed across the whole economy — including to their competitors — and the thing that made them special is now available to anyone. So they refuse. This is the strong default today: the winning enterprise AI pitch right now is “we will never train on your data”.

I don’t ultimately find this convincing, for two reasons.

First, the company can sell the moat at a very high price. AI trained on its data (and perhaps also the data of other similar companies) can generate far more value across the economy than the company could ever capture alone, so the AI company can pay well above the company’s expected future profits and still come out ahead — paying, for example, with shares in the AI company itself. The surplus from combining the data with the AI cognitive labour is enormous, and so there’s plenty of room for both sides to come out ahead. Today the surplus is much smaller, so it’s not surprising that the data isn’t sold.

Still, CEOs might refuse to sell the moat simply because they want to run their own business. Or they might believe they could make more money by holding out and extracting rents from their control of a scarce resource (see Against the Coasean Singularity for a strong version of this pessimism and see footnote for discussion of the game theory21).

Second, and more convincingly, data-sharing deals can be structured to preserve the moat. The AI is fine-tuned on the company’s data under an exclusive arrangement — the resulting AI works in their workflows and nowhere else. Their moat is now embodied in an AI rather than in their employees. Deals like this leave both sides better off, so I expect them to happen. Of course, learning will be slower than if data were pooled across many firms.

Even this deal might fail though. People may not trust the AI company to keep to the agreement. Human employees may strongly resist their own replacement. Legal barriers may delay certain uses of customer data (though I’m sceptical, see footnote22). And human-run companies may just be slow to adopt new technologies, as is common.

But in competitive industries this situation is fragile: once one company in the sector has adopted AI, it will be very hard for others to hold out.

Overall, I’d guess that, in competitive industries, AI will learn to automate most work within a year of the SIE.

Conclusion

Here’s a summary of my views:

So my overall view is:

• Data bottlenecks might significantly delay automating AI R&D — this depends on how well RL generalises as we scale it up and, if it generalises badly, how hard it is to gather lots of data specialised to AI R&D tasks.

• Data bottlenecks are unlikely to stop a software intelligence explosion from accelerating over time. But there will likely be some slowdowns from human-derived training data losing its relevance as AI software advances, and from it becoming harder to increase data quality once it reaches the human-quality ceiling. I expect these slowdowns to be fairly minor: I’d be surprised if they make the SIE take >2x as long.

• After the SIE, I expect AI sample efficiency on real-world tasks to match humans. And in competitive industries, I expect them to rapidly have access to the real-world data needed to automate the work.

Acknowledgements: thanks for comments from Herbie Bradley, Daniel Carey, Alan Chan, Josh Clymer, Owen Cotton-Barratt, Tom Cunningham, Daniel Eth, Lukas Finnveden, Ryan Greenblatt, Brendan Halstead, Anson Ho, Eli Lifland, Alex Mallen, Sam Manning, Sören Mindermann, Dwarkesh Patel, Carl Shulman, James Tillman, and Philip Trammell.

This article was created by Forethought. See the original on our website.

Matthew Adelstein was a Visiting Scholar at Forethought and blogs as Bentham’s Bulldog, where he writes on ethics, animal welfare, and the long-run future.

He joined Fin Moorhouse to discuss:

• Why Matthew thinks handoff of high-stakes decision-making to AI is the most likely route to a great future

• Why locking in current human values would be a cosmic-scale catastrophe, why steering nowhere in particular is barely better, and why the alternative is building AIs that we trust to do reflective moral philosophy

• Various ways handoff might manifest politically: elected AI representatives (“President Claude”), extending political rights to digital minds, or de facto handoff through gradual disempowerment of humanity

• Whether handoff should be reversible (such that humans could take back control if we wanted) or if we should “tie ourselves to the mast”

• Whether to task AIs with “do what’s objectively good” or “do what humans would want on reflection,” why Matthew leans toward the former, and why he expects the two to ultimately converge

• If digital minds end up vastly outnumbering humans, and those digital minds deserve political rights, then consistently applied liberal-democratic principles may imply handoff as reducing expected disenfranchisement

• Whether AIs can actually do philosophy: where models currently sit, bootstrapping schemes using AI-designed evals, training AIs with different epistemic constitutions and looking for convergence, and Fin’s scepticism that recursive self-evaluation works as well as it needs to

• Whether we could even tell if an AI were superhuman at philosophy, given the difficulty of evaluating philosophy that’s beyond our own level of competence

• Whether goodness can compete under evolutionary pressure, resource races into space, and intergalactic existential risks as an argument for top-down planning

• Whether humans’ selfish preferences would saturate in a post-AGI world

• Why smart AIs won’t necessarily converge on good values, and how selection against “weird” outputs (by humans’ lights) to moral reasoning could lock in something mediocre

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

One utopian vision for space expansion involves a libertarian explosion of diversity, with everyone and their robot setting up different sorts of societies and governance structures.1 We might call such a future ‘galactic anarchy’.

On Earth, while there is anarchy between nation states, within a country the government enforces contracts, protects property rights, and provides for collective defence against external aggressors. A government that does nothing else (e.g. provide a social safety net) is Nozick’s ‘nightwatchman state’. Without a Leviathan to uphold the rule of law, we are left with Hobbes’s war of all against all.

But in the cosmic future, time and space make it far harder to have a centralised Leviathan punishing violations ex post. Suppose an aggressor conquers an already-occupied galaxy. It may take millions of years for nearby galaxies to learn about this infringement and prepare a response. And whose job is it anyway to go and punish the aggressors? That might be difficult and costly, and all the nearby galaxies prefer to mind their own business while strengthening their defences. Any inter-galactic governance system that relies on observing and punishing wrongdoing across millions of light-years is doomed to failure. Justice must be swift. And swift justice must be local,2 not from the next galaxy over.

If space is defence dominant, each star (or galaxy) can self-govern in peace without much fear of being conquered or destroyed by a neighbour. Even so, there are three huge negative externalities that a single rogue star system could impose on the rest of the galactic community:

• Unconstrained expansion. In a free-for-all—where anyone can use whatever resources they can grab and defend—people could be incentivised to race as fast as possible to be the first to lay claim to as many resources as possible, even if this involves ‘burning the cosmic commons’ by wasting resources to accelerate more and more probes closer to lightspeed. It’s pretty unclear how significant this waste would be—how feasible would it actually be to quickly harvest most of the energy from a star to speed up probes or launch even more of them?—but in the worst case, this could eat up a lot of the possible value and favour the spread of ‘locust’ value systems that only want to acquire and consume resources.

• Galactic X-risks. Jordan Stone catalogues many (scientifically uncertain) possible ways that a reckless or omnicidal civilisation could drag all others in its lightcone down with it (the ‘vulnerable universe hypothesis’). For instance, it may be possible to trigger false vacuum decay, destroying all particles in a bubble expanding at light speed.3

• Hopefully physics does not allow for this. But it is possible that at technological maturity, rogue star systems or galaxies could destroy the universe. If so, retroactive enforcement and punishment won’t suffice.

• Suffering risks. A misguided or malevolent civilisation could create astronomical amounts of suffering, many orders of magnitude worse than e.g. factory farming on Earth. Given defence dominance, the rest of the universe may be powerless to intervene and punish, or even stop, this.

Even if the chance of any given star system going off the rails in one of these ways is very low, across billions of galaxies each containing millions of star systems that are slowly diverging from their parent civilisations through cultural evolution and drift, it is a statistical inevitability.4

There are also some big upsides that might be harder to achieve under galactic anarchy, such as long-distance moral trade. Suppose that Earl on Earth and Andy in the Andromeda galaxy agree that Earl will spend some resources on hedonium and Andy will spend some resources running copies of Earl. It will plausibly be difficult by default to verify that counterparties are holding up their end of the bargain (from space, compute running hedonium plausibly looks about the same as compute running Earl uploads, so it might be hard for Earl to be confident that Andy is actually running copies of him and not hedonium).

And, even with verification, it might be difficult to enforce contracts from a distance, especially if there are time lags that make it slower to do a tit-for-tat strategy. Suppose that Earl received word that Andy was not running Earl-uploads, 2 million years later. Earl has been faithfully running hedonium for the past 2 million years, so even if Earl stops now, Andy has gotten a lot of value out of Earl. And it might take even longer to propagate the news that Andy is a cheat to all of Andy’s (potential) trade partners.

Funding public goods is also difficult under galactic anarchy. Currently, these are funded by governments coercively taxing their populations and then voting on how to spend the tax dollars. It’s possible that there will be galactic-scale public goods—scientific research, defence against aliens, or setting aside resources for pursuing impartial moral purposes—and the coercive taxation model might be the best way to fund them.

What then can we do? One solution5 is mentioned in passing in AI 2040’s space governance supplement:6

One way to do this [enforce property rights] might be to require that all probes sent to colonize other star systems carry a nightwatchman ASI to prevent the colony from sending out probes to seize space resources that belong to others.

Given the vast distances of space, the monitoring and enforcement governance regime must be dispersed throughout the inhabited universe, rather than in a distant galactic capital. We need to prevent violations before they occur, since we can’t punish them ex post.

Every single inhabited star system should have an unchallengeable governance system that can with 100% reliability7 enforce the universal code of respecting property rights, not destroying the universe, and not creating astronomical suffering. Perfect reliability is not achievable today, but computational systems with backups, error-correction, and formally verified software systems may allow this.

For concreteness, here’s a sketch of a proposal:

• Every probe leaving the solar system would be required to carry an ASI nightwatchman.8

• • This might entail delaying when probes are sent out, so that the design and ruleset of the nightwatchman can be decided upon after sufficient reflection. It may also be desirable to delay space expansion until we are near technological maturity, such that we know better what risks the nightwatchmen must defend against.

• The nightwatchman would maintain a decisive strategic advantage in the colony established by the probe. This might require that the nightwatchman monitor industrial build-up in the system and the creation of new ASIs within the system, to ensure that they are either aligned to the nightwatchman or lack the ability to interfere with the nightwatchman.

• The nightwatchman would prevent probes from leaving that star system unless they also carry a copy of the nightwatchman. It would also prevent probes leaving the system that violate any anti-racing rules that have been set to avoid burning resources wastefully.

• The nightwatchman would prevent anyone in the colony from carrying out any prohibited activities. This probably requires that the nightwatchman or a trusted delegate be able to audit how compute and other resources are being used.9

• Optionally, when making voluntary agreements with people in other star systems, people can make such contracts binding by specifying that the nightwatchman will monitor and enforce the contract. This allows positive-sum trades to be made that otherwise could not be.

• (Maybe) The nightwatchman collects taxes and spends them on public goods.

• The nightwatchman should receive updates to its policies from legitimate authorities.10

  • E.g. if everyone votes that a new prohibition be added to the list,11 then the nightwatchmen in every star system change their policy in response.

  • E.g. if someone discovers a new way of destroying the universe, then this information should be shared to all the nightwatchmen so that they know to monitor for and prevent it.12

We’re obviously leaving a lot of governance questions unanswered—how people decide on prohibitions, how people choose which public goods get funded, etc. The space governance supplement has a bit more discussion on some possible answers to these questions. The ‘nightwatchman’ proposal is intended to be a platform on which many different governance systems could be built.

This proposal has some drawbacks. Here are a few of the most important ones.

It’s a lock-in event, and it would be bad if we locked in a bad policy for the nightwatchmen. If the code is too expansive, and too hard to update, it may close off moral and social progress and innovation. We are certainly glad that no government has historically had the power to lock in a legal code indefinitely. But if the code is too minimal, or too easy to update, rogue actors will find a way to route around the letter of the law, or simply alter the code in undesired ways.

The AI Futures Project’s space governance appendix discusses some possibilities for how the code could be chosen and revised (see the section titled “Mitigating downside risks”). In any case, we think that the final version of the code—if there is a final one—should only be decided after careful reflection.

That said, the nightwatchman proposal tries to preserve as much future human agency as possible without imperilling galactic civilisation. Alternate proposals could give up altogether on diversity (e.g. a ‘hedonium shockwave’) or freedom (e.g. a sovereign AI that is immensely wise and makes all important decisions in each star system). So this may be close to the minimum amount of lock-in necessary to avoid catastrophe.

It introduces a single point of vulnerability among all the colonies. If all the nightwatchmen were identical and it turned out that they had a bug, then we would be in trouble because they have a decisive strategic advantage over all the star systems we’ve settled. We might be able to mitigate this by designing different nightwatchmen in very different ways, even if they would all ultimately be enforcing the same policy. But the more different nightwatchmen designs there are, the higher the chance that one of them is flawed, which allows a star system to go rogue and destroy its lightcone. So there is some inherent tradeoff between diversity and security here.

Plausibly it inherently reduces the value of the future for some people (e.g. it is incompatible with some conceptions of freedom, self-determination, or privacy). Surveillance would probably have to be fairly extensive—the nightwatchman needs to detect attempts to create superweapons, to send spacecraft to other star systems, and to remove the nightwatchman’s ability to monitor and enforce its policies in the future. It’s plausible that it would need to monitor every large industrial facility in each star system. The details will be determined by what is possible at technological maturity, such as the minimum requirements to build and launch a near-light-speed probe.

But enforcement should and could be narrowly targeted to preventing these very serious, galactic-scale harms. Otherwise, the nightwatchman should allow the inhabitants of the star system to govern themselves in whatever way they wish. The universal code could also specify different levels of verification and enforcement: if surveillance is lax enough that a few people avoid paying their galactic taxes to contribute to moral public goods, that is not the end of the universe. But surveillance to prevent false vacuum decay must be incredibly robust.

The nightwatchmen could also be time-limited. After all reachable star systems are settled, and ideally once galaxies have drifted apart such that they are no longer causally connected, the nightwatchmen could switch themselves off. A massive governance failure in one galaxy (e.g. vacuum decay) then could not spread to others, containing the damage. On plausible cosmological theories, the vast majority of lives might be lived in this ‘twilight’ of many causally disconnected pockets. So even if extensive surveillance ends up having large costs, only a very small fraction of lives need be lived under such conditions.13

This would not prevent suffering risks, though.

Currently, we shouldn’t trust any government with this power, as surveillance tech designed to stop terrorists is dual-use and could also be used to e.g. detect political dissidents. But it will likely eventually be possible to build provably secure AI systems with precisely specified remits, in this case to enforce the universal code (see proposals on ‘structured transparency’ for a sketch of what this could look like).

This is a crux for us—if it’s not possible to get these strong guarantees that the nightwatchman would only enforce the narrow universal code, then we would probably oppose it.

Many questions of optimal space governance—such as what combination of democracy, markets, and wise ASI oracles should determine how space resources are used14—remain very open. But we think the question of whether colonists should be allowed to set off into the unknown and entirely self-govern can be answered in the negative. Encouragingly, we expect people will come to realise this before it is too late and the von Neumann probes have left the solar system. But this isn’t guaranteed! This post is our small attempt to make galactic anarchy less likely.

This article was created by Forethought. See the original on our website.

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don’t lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

Keywords: recursive self-improvement, RSI, intelligence explosion, explosive growth, finite time singularity, generation time.

Preview of Figure 1. A vertical asymptote requires the gradient (rise over run) to approach infinity within a finite time. Decreasing the run is key. It cannot be achieved with a fixed feedback generation time (centre) no matter how quickly the improvements grow, but can occur when the generation time approaches zero (right) even with a fixed additive improvement.

The Possibility of an Intelligence Explosion

AI can be applied to automate many different things. One of those is the R&D that goes into building better AI systems. There have already been minor examples of partially automating this process, such as using AI to: find better optimisation algorithms for training neural networks (Andrychowicz et al. 2016), design a more efficient matrix multiply algorithm (Fawzi et al. 2022), help write the code for new AI systems (Anthropic 2026), and run experiments to iteratively improve AI systems (Karpathy 2026). Leading AI companies are increasingly talking about using an AI system to do more and more of the work of designing and building its successor — something they hope (and fear) could lead to a radical acceleration in the rate of progress in AI (Hassabis et al. 2026).

I. J. Good (1965) introduced the idea of AI systems increasing their own intelligence in what he called an ‘intelligence explosion’:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind...

Implicit in Good’s intelligence explosion is the idea that this process of designing a better machine can be iterated. AI_1​ can create AI_2​, which can create AI₃​, and so on. Each system is getting better at intellectual activities — including that of AI R&D — such that each successive system should become ever more intellectually capable.

Following the recent literature, I’ll refer to this process as recursive self-improvement, or RSI (Yudkowsky 2001). I use this term very inclusively, to cover situations where humans are still doing almost all of the work through to situations where AI is doing most of the work — or even all of the work. I include cases where the AI is taking its own files as input and improving them (self-improvement), as well as cases where it is building an entirely new AI system. And I include cases where the intelligence level of successive systems ‘explodes’, as well as cases where it merely improves the speed of progress by some multiple, or where the gains quickly fizzle out.

By accelerating the already rapid progress in AI capabilities, RSI might be extremely dangerous. One reason is that it would likely speed up the risk-creating processes relative to risk-reducing ones (Vaintrob & Cotton-Barratt 2025). For example it may speed up the development of AI capabilities relative to: technical AI safety research, deliberation at the company, and society’s ability to understand and respond. RSI could also provide the opportunity for a single unaligned AI to deliberately corrupt all subsequent AIs. And by enabling larger jumps in capabilities per model release, RSI would remove any opportunity society has to learn from ill effects of intermediately powerful AIs. Finally, by widening the capability gap between a leading AI system and a rival that is a few months behind, RSI would increase the likelihood of a winner-take-all dynamic.

These could provide strong reasons against pursuing RSI. This might warrant legislation restricting it, industry best practices restricting it, or internal company policies restricting it. Such restrictions could take many forms including: careful monitoring, retaining meaningful human control, speed limits on capability increase per month, enforced pauses when various capability levels are reached, or bans on certain approaches to RSI.

My focus here, however, is not on risks (nor responses to risks), but on understanding the dynamics of RSI. In particular I aim to improve our understanding of the qualitatively different kinds of explosive growth that could occur, what conditions produce them, and how likely they are to be possible. While it is entirely possible that RSI might lead to interesting and important growth in AI capabilities that fall short of explosive progress, my focus in this paper is on better understanding the most extreme kinds of growth.

I’ll show:

• how a common approach to modelling RSI conflates two very different kinds of explosive growth,

• how consideration of the physical duration of the feedback loop (the generation time) is key,

• how a super-exponential period of progress would likely end when this generation time can no longer be reduced,

• and how this makes growth with a vertical asymptote somewhat less likely.

Modelling RSI through Differential Equations

A number of recent articles have attempted to mathematically model the possibility of explosive growth from RSI (Aghion et al. 2017, Erdil et al. 2024, Eth & Davidson 2025, Kokotajlo & Lifland 2025). A common approach is to use differential equations connecting the rate at which the AI’s capability increases to its current level of capability. This captures the idea that the amount by which a system can improve its capability in the next time period depends on its current level of capability. While differential equations aren’t the only way to model RSI, they are a logical choice, being the standard way to model feedback loops in physics, engineering, and economics.

Let’s define A to be some measure of the cognitive capability of an AI system (we’ll say more about which measure later). Adopting the Newtonian notation for differential equations, we use Ȧ to represent the derivative of A with respect to time. This is a more compact notation for dA/dt — the rate at which A is changing per unit time.

Let’s start with what may be the most well-known differential equation of all:1

This says that the rate of change of A is in direct proportion to its size. For example, a simple model for the rate of change of a population of animals is that it is in direct proportion to the number of animals, with k set by the difference between the birth rate and the death rate. Solving this differential equation to find how A evolves as a function of time famously gives exponential growth (starting from some initial population, A₀​):

But what if Ȧ doesn’t vary in direct proportion to A? What if there are increasing returns, such that doubling A more than doubles Ȧ? Or decreasing returns, such that doubling A less than doubles Ȧ?

Economists typically allow for this by raising A to some power, r.

This is a flexible approach, covering many different regimes of growth. When r = 1, this is the same equation as before, producing exponential growth. When r = 0 there is linear growth. When r < 0 growth is sub-linear. When 0 < r < 1, A(t) grows faster than linear, but slower than exponential (e.g. r = ½ gives quadratic growth). And finally, when r > 1, A(t) grows more quickly than an exponential. In particular, it grows as a hyperbola, reaching a vertical asymptote at some finite time t*.

This means that as time increases towards t*, A(t) grows explosively, with every finite level of A being exceeded prior to time t*. In such cases it is said that A(t) has a mathematical singularity at t*. This term ‘singularity’ has been adopted by futurists to refer to various kinds of pivotal moment in the future of technology — often with an almost mystical undertone. But for the purposes of this essay, it simply means that the mathematical model of how A changes with time has a vertical asymptote at some particular future time t*.

Economists often use formulas like (3) in endogenous growth theory. This typically involves a set of differential equations connecting total economic output (Y), labour/population (L), capital (K), and productivity/technology/knowledge/ideas (A).

For example, Kremer (1993) looked at the very long run history of economic growth and population, finding that the population growth rate and economic growth rate have increased substantially over the last million years, with the growth rate being roughly proportional to the size of the population at that time. He modelled this with a set of differential equations and found that they produced hyperbolic growth of population and productivity.

In such models, Ȧ is typically expressed as a product of several factors, one of which is A raised to a power. While this power is often restricted to be ≤ 1, there is usually another factor (such as L) which also grows as a function of A. Once the full set of differential equations is solved, we sometimes see that A is effectively raised to a power greater than 1, allowing hyperbolic growth and its finite time singularity.

Several recent models of RSI start with semi-endogenous growth theory. This is a theory introduced by Jones (1995) in which explosive growth is harder to achieve due to diminishing returns in the growth of ideas. Its central equation is:

Where: δ is the productivity per researcher; λ (≤ 1) represents the stepping-on-toes effect, where having twice as many researchers at the same time is less than twice as productive due to issues of duplication and coordination; and β (> 0) represents the fishing-out effect, where subsequent ideas get harder to find.

Holding population (L) constant, this would prevent even exponential growth in technology (A), let alone a singularity. In Jones’s original paper he suggests that while the direct effect of A on Ȧ has diminishing returns, it has also allowed an exponential increase in population, and it is this that drives the exponential rise in technology.

Recent work applying this to RSI has often focused more on improving the efficiency of AI than on improving its intelligence. For example, Davidson et al. (2026) model a situation where AI is able to perform R&D as well as a human and let AA be a measure of its computational efficiency. This sidesteps extremely thorny issues of how to measure intelligence and opens up a clever way of getting hyperbolic growth out of Jones’s model.2 The key is that the total amount of AI labour will be the product of the total compute devoted to RSI (C) multiplied by the AI’s computational efficiency:3

We can then plug this back into equation (4) getting:

Now A is being raised to the power of 1 + λ − β, which could be greater than 1, so could produce hyperbolic growth. Indeed, if we assume that the growth in compute will be much slower than this self-reinforcing growth in efficiency (so effectively hold C constant), and we simplify the equation by defining r to be 1 + λ − β, then we have an equation with exactly the original form of (3). So even in the relatively tame semi-endogenous growth theory, the possibility of a singularity is back on the table.

(It is worth noting that by defining A as a measure of efficiency, this isn’t really a model of an intelligence explosion at all. It is an efficiency explosion. Or more precisely, a labour explosion. This makes the model something of a lower bound on what might happen, since at every point AI also has the option of doing R&D to increase its intelligence and presumably the optimal path involves both efficiency and intelligence improvements. Even if this is a useful lower bound, it does mean there remains an important gap of actually modelling an intelligence explosion.)

As well as the recent flurry of economics-inspired papers, there is a little-known earlier literature by computer scientists. Figures such as Solomonoff (1985), Kurzweil (2001), and Moravec (1999, 2003) also used sets of differential equations to model the dynamics of an intelligence explosion. One key difference is allowing a greater flexibility of functional forms for the relationships. See Sandberg (2013) for an excellent survey.

A Note on Singularities

Before we look into modifying this standard approach, I want to clarify something important about models like these that involve singularities.

While the model has an important quantity (A) rising to infinity in finite time, I’m not sure whether anyone working in this field believes this will actually happen. Instead, they typically believe that the model will cease to match reality at some point before t*.

One reason for this is that there may be some upper limit to how high A can go — either a conceptual limit or a practical limit. For instance, we know that exponential growth is often a good model of the start of a compounding process, but eventually runs into some limiting factor. When we zoom out, we see that it was really just the beginning of a larger S-curve, plateauing at some finite size. In such cases, we can say that the early part of the curve was approximated very closely by an exponential, but there was really some additional term in the equation (for an effect like overcrowding) which started very small, but eventually came to dominate the long-term behaviour.4

This could easily be true for intelligence explosions too. If so, then hyperbolic growth might fit the true trajectory of A very closely until it gets into the vicinity of some upper bound A*, where it starts to fit more and more poorly. Proponents of these models are merely saying that there could be some substantial part of the trajectory of A where the model is a good fit — better than equally simple alternatives such as exponentials (Sandberg 2013). If A only plateaus at some point many orders of magnitude above its starting point, then even if the hyperbola only fits for a small domain of time, it might fit for a very large range of capabilities. If it predicts those much better than alternatives — and explains why things are increasing so fast — it would count as a successful model. It’s OK for a model to only have a finite domain of applicability.

Just as exponential growth can be a great model for the start of a process, so can hyperbolic growth. Hyperbolic growth will probably cease to be a good model at an earlier time, though not necessarily at a lower height on the graph.

In physics, singularities in the models of particular systems are often seen as useful pointers to where those models must break down and new (hitherto unmodelled) behaviour must begin. We could also adopt that frame here and see the models as pointing to some finite time t*, before which some new unmodelled aspect of the system must take over.

So we’ll explore the behaviour of mathematical models of RSI — which sometimes include a singularity — but will remember that something will probably stop the real physical process prior to that point. And we will explicitly return to these limitations in the final section.

Generalising the Standard Differential Equation for RSI

If we take a closer look at equation (3) (Ȧ = kA^r) we might wonder why it takes this particular form. Taking an arbitrary power of A is certainly a natural and convenient way to create an adjustable differential equation that allows for both sub-exponential and super-exponential growth. But it is one very specific way of doing it, raising the question of whether some of the results about singularities depend on this particular form. We might ask:

• Are there differential equations that produce intermediate patterns of growth lying between exponential and hyperbolic growth?

• What feature of the differential equation is producing the singularity? Is it super-linearity? Being everywhere convex? Something else?

• Do we still get a singularity if we make slight changes to the function (such as adding a constant, or inducing a slight wobble)?

Let’s find out, starting with the more general form of the first-order autonomous differential equation:

What conditions do we need to impose on ff to get super-exponential growth? Does this then always lead to a singularity?

Super-exponential growth means that A(t) eventually overtakes every exponential. Since exponentials have constant relative growth rate (Ȧ/A = k) we require that this relative growth rate (f(A)/A​) overtakes all constant levels:

In other words, f must grow super-linearly in A. This ensures the right behaviour when A is high enough.

To this we must add a second condition to ensure A can grow high enough:5

at every point in

Let’s call this the positivity condition. Together, these conditions are sufficient for super-exponential growth.

What is required of f for A(t) to possess a singularity? This comes down to whether it takes a finite time for A(t) to approach an infinite height. We can work out the time at which A(t) reaches a given height A′, by starting at A₀​ and adding up the amount of time needed to accumulate each small gain in height dA. The time needed for each gain is just the reciprocal of the slope 1/f(A)​ times dA. So the total time to reach a height A′ is just:

And thus the time it takes to climb all the way from A₀​ to ∞ is:

When this integral converges to a finite value, it means that there is a finite time by which A(t) goes to infinity — i.e. a singularity. The convergence of this integral is the key condition for A(t) having a singularity, which we shall call the blow-up condition. To this we again add the positivity condition — f(A) > 0 at every point in [A₀, ∞) — which ensures A(t) is growing (rather than shrinking) and that there is no division by zero.

In the standard differential equations for RSI (where Ȧ is some power law of A) everything that meets the super-linearity condition also meets the blow-up condition: so everything that is super-exponential has a singularity.

But we can now show that this is not true in general. Consider:

A log⁡(A) is super-linear, so A(t) grows super-exponentially. But it doesn’t meet the blow-up condition: the integral of 1/Alog⁡(A)​ diverges to infinity, so it takes infinitely long for A to grow infinitely large. Solving the differential equation reveals that A in fact grows doubly exponentially with time:

This is super-exponential but has no singularity. So if RSI were to obey this equation, there would be a very rapid rise in capabilities, but one of quite a different kind. It would always have more room for improvement, even after an arbitrarily long period of growth.

And this isn’t the only function that produces such growth. For instance, consider the infinite family of equations:

• Ȧ = Alog⁡(A)

• Ȧ = Alog⁡(A) log⁡(log⁡(A))

• Ȧ = Alog⁡(A) log⁡(log⁡(A)) log⁡(log(log⁡(A)))

• ...

These all fail to meet the blow-up condition, so do not possess singularities. Instead they grow as a double exponential of time, a triple exponential, a quadruple exponential, and so on. Yet they also come extremely close to the blow-up condition. If you raise the final factor in any of them to a power greater than one (e.g. Alog⁡(A)^1+ε), they satisfy the blow-up condition and A(t) will have a singularity.

(See the appendix for a table showing the rates of growth corresponding to a wide range of f(A), including whether they produce singularities.)

So in the general case where Ȧ = f(A), there is a narrow zone of functions for f(A) that fit between the exponential growth given by Ȧ = kA and the growth towards a finite time singularity given by Ȧ = kA^1+ε. We thus can’t assume that all super-exponential growth has a singularity.

We might wonder whether it is realistically possible to end up in this zone. Economists call exponential growth a ‘knife-edge solution’ to Ȧ = kA^r, suggesting that it is vanishingly unlikely for r to take on the value of exactly 1. If so, this new zone I’m pointing to may look like it is on the edge of the edge of the knife. Or it would if we were taking this functional form and supposing a random value of r. But the assumption of the functional form is then doing most of the work. If we instead suppose an unknown functional form for ff, with a non-zero chance of it being Alog⁡(A), the landing zone looks a little larger.

I think this functional form is plausible when the A and the log⁡(A) are coming from different places. For example, suppose A is a measure of the total number of AI researchers doing RSI, and suppose that the contribution of each one towards increasing A is proportional to the log of their total population (due to weak spillovers from each person’s research, or weak economies of scale). Then you get a combined effect where Ȧ = kAlog⁡(A).6

That said, it is still a narrow zone and any kind of fishing-out or stepping-on-toes effect could easily knock things out of the zone. Overall, I think this possibility would be just something of an interesting footnote were it not for new considerations we’ll see in the next section, which substantially widen this zone.

Finally, it is important to note that in this general case of Ȧ = f(A), one can’t treat all growth with a singularity as hyperbolic. That was only true when making the modelling assumption that f(A) has the form of a power law. Some functions (such as Ȧ = Alog⁡(A)²) produce trajectories that grow more quickly than any hyperbola —though they are simultaneously less ‘explosive’ in the sense that the growth isn’t as packed into the final moment. Others (such as Ȧ = e^A) produce trajectories that grow more slowly than any hyperbola, but are more explosive since they still reach infinity in finite time, despite lagging behind until the last moment. To reflect this wider class of functions, I’ll use the more inclusive term singular growth to refer to all cases when A(t) possesses a singularity (whether hyperbolic or not).

We can now answer our earlier questions:

• There are differential equations that produce intermediate patterns of growth lying between exponential and singular growth, such as Ȧ = Alog⁡(A), which gives doubly exponential growth.

• f(A) being super-linear in the limit produces the super-exponentiality, but more is needed to create a singularity. Being everywhere convex is neither necessary nor sufficient — singularities are not produced by satisfying a local property like convexity, but by satisfying a global property where deficient growth in f(A) somewhere can be made up by faster growth elsewhere (so long as it is always positive). The relevant global property is the blow-up condition: that the integral of the reciprocal of f(A) converges.

• Small changes to f, such as small additive or multiplicative constants or small random deviations from its path won’t usually change these behaviours, unless they make f(A) ≤ 0 somewhere along the trajectory.

Feedback loops & discrete time steps

The idea of an intelligence explosion relies on some kind of feedback loop, where each AI system builds a more intelligent successor, which is even better at AI R&D. There are many forms this feedback loop could take. For example:

• AI could develop new tools and processes for making faster chips.

• AI could create better chip designs for the current chip manufacturing processes.

• AI could make the pretraining process for the next model more efficient, getting that model sooner.

• AI could improve the pretraining process for the next model to make the model more capable.

• AI could develop ways to fine-tune its own weights to make it more capable.

• AI could modify its harness to make it more capable.

Something these all have in common is that they don’t happen instantly. Rather than a continuous process, such as the differential equations behind an object cooling or a pendulum swinging, we instead have something whose capability is increasing in discrete steps.

Such discrete steps are quite common in feedback processes. Even the classic example of a microphone near a speaker gives discrete jumps in volume. When the microphone is switched on it takes a moment for its signal to travel down the wire to the speaker, which then jumps up in volume. This louder sound then has to travel through the air at the speed of sound, before the microphone can register this increase and begin the next cycle. If the microphone were 30 metres from the speaker, the volume would rise in a staircase pattern with each step lasting about a tenth of a second. Because these steps are so brief, we often don’t hear them, and continuous models of the feedback are adequate for most purposes.

But the steps in many RSI feedback loops are much longer — potentially months or years. And even more importantly, we’ve suggested that the RSI feedback loops might be able to produce singular growth. Is that even possible in discrete time or just an artefact of the continuous nature of differential equations?

Taking the discrete nature seriously will change how we see the idealised dynamics of these systems and will change the conditions under which singularities are possible. By the time we’re finished, we’ll see the possibility of super-exponential, yet sub-singular growth go from being a narrow possibility to a mainline scenario.

We can start by considering the discrete version of the differential equation: the difference equation. Let A_n​ represent the system’s ability after n times around the feedback loop. The discrete version of Ȧ is ΔA, which is defined as:

In other words, ΔA is the difference between successive terms in a sequence. Our general autonomous differential equation, Ȧ = f(A), becomes:

Or equivalently (via the definition of ΔA):

In difference equations it is impossible to produce a singularity no matter how quickly f grows. This becomes obvious when one looks at what would be required. How could you have A_n​ grow without bound prior to some particular n? That’s only possible if there are infinitely many steps prior to that point or if there is a step where A_n​ takes an infinite value. But there are only finitely many steps prior to any particular n, and all values are finite (they start from a finite level (A₀​) and each one only adds finitely much to the total, since f is a function from ℝ to ℝ).

For example, if you try f(A) = A² (which would give hyperbolic growth using a differential equation) you instead get doubly exponential growth. Or if f(A) = e^A, then A_n​ grows as a tower of exponentials with n levels (tetration). These are very fast rates of growth (clearly super-exponential), yet they have no singularities. For differential equations, rates of growth like these only lived in a very slender zone, but they are entirely generic for difference equations — occurring whenever f is super-linear.

What do these facts about difference equations mean when it comes to RSI? They don’t have any direct meaning until we say how the number of feedback cycles (n) maps onto time (t). Let’s create a new model of RSI which is neither a simple differential equation nor difference equation. Instead it will be a difference equation that is embedded into continuous time, which we shall call a time-embedded difference equation.

Let t_n​ be the time at which the nth feedback loop is completed and let T_n​ be the time it takes to go around the feedback loop for the nth time. This second parameter (T_n​) will turn out to be a key quantity when analysing the dynamics of intelligence explosions. I shall call it the generation time, via analogy to the parameter of that name in the feedback loops of population growth, disease spread, and nuclear chain reactions. In those subjects, every person/infection/fission produces a random whole number of new persons/infections/fissions, and the generation time is the average amount of time this takes.7 That corresponds to the amount of time it takes to go around their feedback loops, so I’ll use the term ‘generation time’ even though for some RSI feedback loops there won’t be clear discrete generations of AIs.

T_n​ and t_n​ are related by the equations:8

and

Let’s also define n_t​ to be the number of cycles around the feedback loop that have been completed by time t (the highest n such that t_n ≤ t). We can then represent a time-embedded difference equation by the pair of sequences A_n​ and t_n​. Together, these enable us to see how A changes over time, via:

If every cycle of the feedback loop takes the same time, we’d have T_n = k and tn = kn​. If so, the previous remarks about difference equations would apply directly — there are only finitely many loops between any two times, each of which makes only a finite change to A, making singularities impossible.

The same is true whenever there is a bound on how short the loop can get. If it never gets shorter than T_min⁡​ then the growth of A(t) is bounded above by that of a feedback loop with a constant generation time T_min⁡.

But what if the generation time decreases towards zero? If it decreases slowly, such as via T_n = 1/n​, then the total time required to perform n loops increases without bound. This implies there are only a finite number of loops between any two times, making singularities impossible. But what if the generation time decreases more quickly? Let’s call the time taken to perform infinitely many steps t_∞​:

This can be finite. For example, if the generation time shrinks exponentially as T_n = 1/(2n), then t_∞ = 1. In this case an infinite number of loops have been performed within 1 unit of time. As long as A_n​ increases without bound, this would be a singularity at t = 1. So singularities are possible in this model of RSI via a difference equation embedded in time.

What is required to produce them?

First, let’s look at the generation time. It has to go to zero fast enough for the infinite sum in (20) to converge. T_n = 1/n​ is too slow, while Tn = 1/(n^1+ε)​ is fast enough. The cut-off for how quickly the denominator of this fraction must grow turns out to be exactly the same as for the blow-up condition we examined earlier. For example,

is slightly too slow to allow for a singularity, while raising the final factor to any power greater than 1 is sufficient to force the sum to converge, giving a finite t_∞​ and thus the opportunity for a singularity. Let’s call this condition on how quickly T_n​ must converge to allow a singularity the Zeno condition on T_n​, as it is precisely the condition for when infinitely many steps can be performed in a finite time.

What about the rate of growth of A_n? We’ve already seen that no growth rate can give a singularity if we only go around the feedback loop finitely many times, but if we get to go around it infinitely many times, then even adding one each time (f(A_n) = 1) is enough to produce singular growth. Indeed, even adding smaller and smaller amounts each time around the loop can work, so long as A_n​ grows without bound. Let’s call this the boundlessness condition on A_n​.

For example, if we added just 1/n​ after the nth feedback loop, that would be enough to produce unbounded growth and (if the Zeno condition is also met) a singularity. The same is true if we added only

or

​Considered as a function of n, the amount that needs to be added each time (ΔA_n​) is thus set by exactly the same threshold as T_n​ (though ΔA_n​ mustn’t shrink more quickly than this threshold, while T_n​ must). We can summarise this with the following theorem.

Singularity Theorem:

When the growth of A is governed by time-embedded difference equations:

A(t) has a singularity iff T_n meets the Zeno condition & A_n​ meets the boundlessness condition.

Or equivalently:

A(t) has a singularity iff ∑T_n​ converges & ∑ΔA_n​ diverges to +∞

Proof: Both conditions are necessary because it doesn’t help to grow boundlessly if you can only get through finitely many steps by any particular time, and it doesn’t help to have infinitely many steps before some time if the function has a finite bound. But if both conditions are met, then infinitely many steps will happen by t_∞​ and these will take A_n​ beyond every finite level prior to that time.

This theorem cleanly cuts the requirements into two independent thresholds. Success on each is binary: no deficiency on either can be made up by over-performance on the other. We can get a feel for how sharp this is through a pair of examples.

First, consider a feedback loop whose generation time decreases towards zero as 1/n​ and each time it goes through the loop, A gets multiplied by 2^A. A_n​ is growing extraordinarily quickly (tetrationally), but because the generation time doesn’t meet the Zeno condition, there is no singularity. Second, consider a feedback loop whose generation time decreases towards zero slightly more quickly (as 1/n^1.01​), but where only a small and diminishing amount is added to A each time (+1/n​). This meets both conditions, so blows up to a singularity. The tiny difference in how generation times decreased with n outweighed the radical difference in what happened in each loop.

It is very interesting that in this discrete setting singular growth requires generation time to go to zero, and the rate at which it needs to approach zero is analogous to the rate at which f(A) had to grow in the continuous setting. This suggests that the decreasing generation time is driving the whole effect. One way to see this is that singularities require the gradient of the curve to go to infinity in finite time, and there are two ways to increase gradient: increasing the rise or decreasing the run.

In continuous models, there is no way to distinguish these. In raw difference equations, one cannot reduce the run and no amount of growth in the rise can produce a singularity. But when we allow for variable-length discrete time steps through a time-embedded difference equation, we see that it is almost entirely about reducing the run. The rise per step doesn’t have to increase at all — its only constraint is that it can’t decrease too quickly.

Another lesson is that a singularity requires the growth of A to be infinitely compounded. We are familiar with the way compounding interest daily leads to a higher growth rate than compounding it annually. Compounding at ever finer intervals converges towards the growth rate set by continuous compounding. But for super-exponential growth, the value of compounding is much greater. If the benefits can only compound finitely many times by a finite point in time, there is no way to achieve a singularity. It is a necessary condition for a singularity that the growth in A is compounded infinitely many times in a finite time interval — which can happen either via a convergent sequence of discrete generation times or a continuous system.

Solomonoff (1985) provides a nice toy model for an intelligence explosion via shortening the generation time. Computing speeds have been improving exponentially over time (a version of Moore’s Law). If we reached a point where AI could provide the labour needed to keep Moore’s Law running, then every time Moore’s Law doubles computing speeds, it doubles the speed of these digital researchers, halving the time it takes for the next doubling of speeds. So if the first doubling took 2 years, the second would take 1 year, the third half a year... and by the end of 4 years, there would be a singularity — with infinite AI labour and infinite computational efficiency.

Moravec (1999, 2003) developed an improved version of this model, showing that it doesn’t even require Moore’s Law to be exponential. He first noticed that a quadratic speed-up would suffice, while a linear speed-up would not. Then he found his way to the peculiar threshold we’ve seen so many times: so long as the nth generation machines are faster than the first generation by a factor greater than nlog⁡(n)log⁡(log⁡(n))… this approach produces a singularity.

Of course, there are still great obstacles making this impractical. Moore’s Law has required increasing amounts of labour to keep it going (due to fishing-out and stepping-on-toes effects), as well as increasing amounts of capital. Moreover, there will be ultimate physical limits to how far it can go, and there are practical limits to how quickly a new generation of faster chips can be produced (especially when a novel method is required). But it is still an instructive model for an efficiency-driven intelligence explosion. While Solomonoff and Moravec modelled it with differential equations, the presence of an explicit generation time that tends towards zero allows this model to work just as well in time-embedded difference equations.

Eth and Davidson (2025) considered something like this when they explored AI systems that improve their own training time. Suppose an AI system could make sufficient tweaks to its architecture and training process that it slightly improved the speed of training and inference for the next generation of AI systems. If this could be repeated indefinitely, then the generation time (to design and train the next generation) would keep falling. So long as the speed improvements are above the familiar threshold (nlog⁡(n)log⁡(log⁡(n))…), one would get singular growth, leading to unlimited AI labour by some finite time. Of course, Eth and Davidson don’t suggest that we can actually make an indefinite series of such improvements. Instead, one would expect the generation time for training the next generation of models to bottom out at some unyielding finite limit, prematurely ending the period of singular growth.

In general, it seems highly unlikely that generation times can be brought arbitrarily close to zero. This provides an important kind of barrier to singular growth.

It is essential in all these arguments that we are measuring generation time itself, and not the related concept of doubling time:

Doubling time is often useful as it takes into account how much a feedback loop contributes as well as how long it takes. For instance, a loop that takes one second and doubles its input has the same doubling time as one that takes two seconds and quadruples its input. A key advantage of doubling time is that it is also defined for continuous processes, while generation time is not.

However, this combining of the loop’s duration and impact turns out to be doubling time’s downfall. If A_n​ grows very quickly (e.g. doubly exponentially) while the generation time stays constant, then the doubling time will rapidly decline to zero. But we’ve seen that since generation time is constant, there can’t be singular growth no matter how quickly A_n​ grows or D_n​ shrinks. Doubling time shrinking towards zero is a useful threshold for defining super-exponential growth, but it is only generation time that can set the threshold for singular growth.

It is worth noting that one could also model the length of feedback loops via delayed differential equations rather than difference equations. In the equation Ȧ = f(A) we’d been suppressing the dependence on time. We could have equally well written Ȧ(t) = f(A(t)). We can change this to a delayed differential equation where the current value of Ȧ depends on the value of A at a time T units earlier (to account for the generation time): Ȧ(t) = f(A(t−T)). Like the difference equation, this delayed differential equation with a fixed delay can produce super-exponential growth but can’t produce a singularity.

And we can make an analogue to the time-embedded difference equation, by allowing the feedback loop length to change with time: Ȧ(t) = f(A(t−T(t))). Like the time-embedded difference equation, this general delayed differential equation can produce a singularity, but only when T shrinks towards zero sufficiently quickly as t → t*. This gives a continuous model with similar dynamics. I find it slightly less accurate (losing track of the discrete nature of the feedback loop) and slightly harder to work with, but other researchers may find it useful.

We are now able to zoom out and take another look at when growth from feedback loops is singular versus merely super-exponential:

• For differential equations of the common restricted form, Ȧ = kA^r, all super-exponential growth is singular growth.

• For differential equations of the more general form, Ȧ = f(A), we see that there is also a narrow range of super-exponential growth without a singularity.

• When we take the discrete nature of feedback loops into account with difference equations of the form ΔA_n = f(A_n), all super-exponential growth is sub-singular.

• When we allow for changing generation times using time-embedded difference equations, we can again get super-exponential growth in singular and sub-singular varieties. But compared with the setting of differential equations, singular growth is now much harder to achieve due to the requirement that the length of each feedback loop shrinks towards zero. Super-exponential growth without a singularity is no longer a curiosity — it is the default.

Intelligence Measures

When using a differential or difference equation to model RSI, it is remarkably unclear what A should represent. Should it represent intelligence, or should we sidestep that and measure the computational efficiency of that intelligence? If we choose to represent intelligence itself (and thus try to model an actual intelligence explosion), we run into two further problems.

First, it is widely recognised that there is no generally agreed conception of intelligence to measure. Even among those who agree that intelligence is a real and important thing, there is no consensus on which thing it is.

AI research tries to sidestep this by measuring a wide variety of different capabilities via benchmarks. These measure what fraction of a set of tasks related to that capability the system can successfully solve. There is a common feeling that achieving artificial general intelligence (AGI) will require roughly human performance on most such capabilities, but there is some dissent. For example Chollet (2019) argues that intelligence is really the ability to efficiently learn a wide variety of capabilities. On his view, intelligence isn’t measured in terms of capability, but capability per unit input (where inputs could include training data, information embedded in the priors, training compute, inference compute etc.). Even if there were agreement on whether intelligence is a measure of capability or capability per unit input, one would need further agreement on how to weight the different kinds of capabilities into an overall linear ordering for A.

There is also a second problem, which is much less widely recognised. Even if we could agree on the kind of thing being measured (e.g. we could agree on a linear ordering of all AI systems according to intelligence) it is very unclear what cardinal structure this should have — how to assign numbers to those systems.

For example, when measuring the capability of AI systems at playing a game such as chess or Go, researchers often use Elo scores. But Elo scores are really just the log of a more fundamental measure from the Bradley-Terry model which assigns each player a strength such that the odds ratio of a player beating another is simply the ratio of their strengths. Elo is just the log of this strength (with some arbitrary constants to help its numbers match an earlier chess ranking system).

This causes confusion in the literature when papers claim that in contrast to some other domains, chess AI is only improving linearly over time. Chess improvements have been roughly linear when measured in Elo, but the headline claim could equally be that chess-playing AI is improving exponentially over time (when measured in the improvement in the odds-ratio of beating a player of fixed strength). In this case, it is unclear whether progress in chess AI is best described as exponential or linear.

Overall measures for AI progress suffer from the same issue. They could be exponential on one fairly natural scale and linear on another. Or they could be convex (with increasing returns) on one scale while concave (with diminishing returns) on another.9 Given there is often very little discussion or agreement on which measure is the more natural one (or whether there is even a fact of the matter about that) there is a big problem of measure-dependence for attempts to track progress in AI.

How do these issues affect the dynamics of RSI — or our ability to measure and track those dynamics?

Let’s start with a simple example where A is a measure of intelligence and B is another measure where B = log⁡(A). For the differential equation model, where Ȧ = f(A), we will also have Ḃ = g(B), where g is a slower growing function than f. For example, when Ȧ = kA, Ḃ = k. These are consistent (when the measure A(t) grows exponentially, it makes sense that the measure B(t) is growing linearly) but they show that the same rate of progress of the observable phenomenon in the world can correspond to different functions ff in the differential equation.

However, there is an important invariant: f(A) grows fast enough to produce a singularity if and only if g(B) grows fast enough to produce a singularity. i.e. f will meet the blow-up condition (of growing faster than Alog⁡(A)log⁡(log⁡(A)) (etc.) if and only if g also does. It is easy to see this must be true because when some quantity grows without bound, the log of that quantity also grows without bound. So if one grows without bound by time t* the other must too.

Let’s define two measures as similar when either measure growing without bound implies the other does too. When A and B are similar, then either both f and g meet the blow-up condition or neither do. Therefore, if we are trying to detect singular growth of an RSI system in terms of how Ȧ is increasing with A, the threshold is at the same place (roughly Alog⁡(A)log⁡(log⁡(A))...) for a wide range of different ways of measuring that system’s capabilities.10

Something very similar is true if we model RSI via time-embedded difference equations. In this case, the only necessary condition on A_n​ to achieve a singularity is that it grows without bound. And if A_n​ and B_n​ are similar, then they either both satisfy this condition or neither does. So the threshold for how much they need to improve on the nth feedback loop (nlog⁡(n)log⁡(log⁡(n)) ...) is the same for both, and we don’t need to worry about which of these capability measures we are using when testing for a singularity.

But some measures are not similar to each other. For example, when the full range of measure A only maps into a finite interval in measure B, then A going to infinity doesn’t entail B going to infinity and you could get singular growth in A without singular growth in B. This could easily happen if A represented open-ended progress in some domain of intelligence while B was a broader measure that included more domains. In such a case, there would be no contradiction in A having singular growth without B having it too. The time t* would simply be the time at which problems of the domain A was measuring are effectively solved.

This might be the case for measures of A like computational efficiency or ‘effective compute’. Many problems would be solved if we let an AI system have unlimited effective compute (for training and/or inference), but it isn’t clear such a system would excel at all kinds of intellectual tasks. For example, even if you had unlimited compute, it isn’t clear that current architectures and training environments would allow a system to succeed in domains which are lacking a clean algorithmic way of rating the quality of the outputs. Much of the recent progress in AI has relied on reinforcement learning with verifiable rewards (RLVR), but when there are no verifiable rewards, performance may plateau. If so, even a completed singularity in measures like efficiency and effective compute needn’t imply that the system is more capable than a human across the board after time t*.

In physics, scientists distinguish between a coordinate singularity and an essential singularity. The former exists when the singularity is an artefact of how things are being measured. For example, when Schwarzschild (1916) published his mathematical description of a black hole there was a mathematical singularity at the black hole’s event horizon. It took decades of scientific work before physicists were able to establish that this was a mere artefact of the coordinates being used (whereas the singularity at the centre of the black hole was not fixable by a change in coordinates).

We can adopt this distinction when studying RSI. For example, suppose we are measuring A by the mean time between failures (MTBF): how long the system can go before making a mistake while completing tasks of a certain kind. This is a standard measure in engineering and a relevant measure of capability for many real-world uses. In this case, one might see A rising towards a singularity. But consider an alternative measure B which is the percentage of times the system gets the right answer. If A goes to infinity at time t*, that just means B reaches 100% at t*. While a lot of systems never quite reach 100% reliability, there is nothing impossible or paradoxical about doing so. Thus in this example, t* is a coordinate singularity in measure A, but not in measure B, and is not an essential singularity.

Note that a very widely used measure of general AI capability — the METR time horizon (Kwa et al. 2025) — is somewhat like a mean time between failures. Recent years have seen exponential (or perhaps super-exponential) growth in time horizons for AI systems completing the kinds of routine professional computer-use tasks in the benchmark. But even if the time horizon went to infinity,11 it isn’t clear the system would be as intelligent as a human. It may still lack various cognitive skills (such as continual learning, sample efficiency, or creativity), which are important but not being tested. And there may still be substantial room for other AI systems to be much more capable, even at the skills that were being tested — systems that could solve much more challenging computer-use problems using much less compute. An infinite METR time horizon would just mean that the AI system can do the kinds of tasks in the benchmark with perfect reliability (and unbounded duration).

So even though most people studying intelligence explosions assume that the capability measure A won’t actually go to infinity in finite time, it wouldn’t be absurd or unphysical for it to do so. It depends on what measure it is. Some measures could go to infinity without incident. If such measures could also satisfy the key differential equation (or time-embedded difference equation) then one could get a completed intelligence explosion — A would have gone to infinity in finite time.

I’m not quite sure what we should think about this. It really depends on the nature of the measure, and this result is much easier to achieve on narrow measures — those that can rise unboundedly even when many key cognitive abilities are lacking. But measures like mean time between failures or METR time horizons seem unlikely to lead to rapidly decreasing generation times — what goes to infinity is a measure of duration, not a measure of speed. By showing that a completed singularity is theoretically possible, I’m mainly trying to caution people about the behaviour of certain intelligence measures, rather than trying to suggest we will genuinely have unbounded progress in intelligence in a finite time.

Let’s now turn to look at what happens when an intelligence explosion can’t get to infinity.

Going Finite

We’ve now seen how super-exponential growth can be cleanly divided into singular and sub-singular varieties, with the singular kind appearing very difficult to achieve (once we take the discrete nature of feedback loops into account). And we’ve seen that the rates at which f(A) has to grow or diminish are independent of which measure is used (so long as the relevant measures are ‘similar’ to each other).

But all of these claims rely on a clean mathematical model where growth is classified by its asymptotic behaviour as t or A approaches infinity. In the real world, this model may very well break at some finite value of t or A, giving increasingly inaccurate results thereafter. That’s a big problem for this kind of asymptotic analysis — as it relies on the infinite domain (or range) to produce its clean classifications. I hope that these divisions which are natural in the idealised setting will be natural in the real-world application. This is often the case,12 though it isn’t guaranteed.

There are a variety of finite limits we might expect to run into, which could force the clean mathematical model to break at different points, and in different ways.

The most widely discussed is a ceiling on intelligence: a horizontal line at some level A*, which A(t) cannot cross. In the study of feedback systems, people often say that A saturates as it approaches such a horizontal asymptote. It is generally thought that this will happen for RSI.

There are many different kinds of ceiling at which the AI’s intelligence (or efficiency) may saturate:

• Limits of intelligence itself — Even an optimal reasoner would be neither omniscient nor omnipotent. It would need to perform experiments to gain knowledge, could still be beaten at unbalanced games, and may face intractable prediction problems in chaotic or agentic domains.

• Limits of intelligence per unit resource — Even if we have the optimal algorithms and hardware, our solar system has only one star out of the 200,000,000,000 in our galaxy, and growth beyond our system is slow and cubic. If a galactic superintelligence would be more intelligent than a stellar superintelligence, it will be a long time before we could reach that higher level.

• Limits of the hardware paradigm — Even an optimal silicon chip may be far below the physical limits of compute per unit resource.

• Limits of the algorithmic paradigm — Even an optimal neural network may be far below the best intelligence that could be achieved with that amount of compute.

• Limits of training data — The training data we have (and could acquire during RSI) is lacking a lot of information on many domains (especially non-verbalisable information and contextual information). So we could have a very intelligent system that is limited by inability to train certain skills.

• Earlier limits — The ascent may stall out before any of the above due to limitations inherent in the starting AI.

Discussions of RSI sometimes tacitly assume that an intelligence explosion would only stop when it reached the limits of intelligence itself, and then use assumptions like omniscience or perfect rationality to model its behaviour. But an explosion might saturate significantly below this level, with important consequences for predicting post-explosion capabilities and for understanding whether all intelligence explosions need end at the same intelligence level.

The manner in which super-exponential growth slows down as it approaches its limit might be analogous to the way exponential growth runs out of steam. While the simple equation for exponential growth is Ȧ = kA, in reality there is usually an additional dampening term that starts small but grows to dominate for large A. For example if we subtract a quadratic term that has been shrunk down by a large factor (K), this gives the differential equation for logistic growth,

where A(t) approaches a horizontal asymptote at height K. We could use the same model for super-exponential growth:

Or more generally, we could assume Ȧ = f(A) − g(A), where g(A₀) ≈ 0 and g(A) < f(A) until some crossover point, A*. This will produce a horizontal asymptote at height A*.13

This dampening force could come from running out of room for improvement as a system approaches some form of perfection (such as optimal intelligence per unit resource), or it could come from something like straining under the growing size or complexity of the system. The latter could stop the ascent before reaching any kind of optimal system.14

As well as a ceiling on A, one could also run into upper limits on how quickly it can increase — either in absolute terms (such as a limit on Ȧ) or percentage terms (such as a limit on Ȧ/A​). Rather than making A(t) converge to a horizontal asymptote, a maximum gradient would make A(t) converge towards a diagonal line, while a maximum growth rate would make A(t) converge towards an exponential.

What could produce limits of these kinds?

One mechanism I find particularly likely is if there is a floor on the achievable generation time, T_n​. It seems very unlikely that generation times can be brought arbitrarily close to zero. There are many kinds of feedback loop that could contribute to RSI, ranging from decades (e.g. designing a successor to EUV lithography) to months (e.g. designing better pretraining) to seconds (e.g. designing better scaffolds). While some are extremely short, those only capture a tiny fraction of the pipeline of what makes for better AI R&D, so it seems likely they would quickly saturate if the other aspects of AI improvement were unchanged. For example, putting a fixed agent in a scaffold that has the agent repeatedly redesign that very scaffold might make some improvements, but without changing the model itself, it seems very unlikely to take off towards infinity. And while continued RSI might be able to reduce any of these generation times by a substantial factor, each one seems likely to run into constraints where a certain minimal amount of time has to pass in order to make an improvement.

What would a lower bound on the achievable generation time do? If T_n​ approached some minimal value (T*) while A_n​ grew by a fixed amount each loop (ΔA_n = k), then A(t) would approach a linear rate of increase. If A_n​ instead grew by a fixed proportion each loop (ΔA_n = kA_n​), then A(t) would approach an exponential rate of increase.

I think the second of these looks quite plausible. If so, we might see multiple phases of an explosion:

0. The initial exponential phase when the doubling time of A is driven by human-only research.

1. Increasing amounts of RSI drive the generation time down towards machine speeds, so the growth rate of A climbs from its human-only rate towards some very high fully automated rate. Because this phase involves a continuously increasing growth rate, it is super-exponential in shape.15

2. Fully automated RSI continues for a while at this faster exponential, but as A climbs, it starts to saturate. This phase would be exponential in shape.

3. As it approaches its inflection point and subsequent horizontal plateau, the trajectory has substantially departed from its exponential form and is revealed to be a logistic.

This trajectory involves two kinds of finite limits on the idealised process — first on generation time (and thus growth rate of A), then on the absolute level of A. It shows how we could have a super-exponential process (which would be worth modelling as such), yet it runs out of steam in two distinct ways. On this view, the mathematical models of super-exponential growth could be a good fit (and thus predictive) for the period where generation time is reducing towards zero in some clean way (e.g. 10% shorter each loop), but then cease to be a good model as we enter phase 2.

I’ve been assuming that A would be improving by a constant factor each time round the loop, guaranteeing super-exponential and logistic phases, but versions with other functions for f(A_n) are also possible and can still show distinct phases if generation time saturates prior to intelligence level saturating.

Conclusions

My aim in this paper has been to improve our theoretical understanding of intelligence explosions — looking at the most explosive possibilities (super-exponential and singular growth) and asking what conditions are required to produce them.

We first looked at models based on differential equations and saw that the key threshold for super-exponential growth of some measure A is when Ȧ grows superlinearly in A. But the threshold for singular growth is somewhat higher — Ȧ has to grow slightly faster than all functions of the form Alog⁡(A) log⁡(log⁡(A))... This allows for the possibility of ending up with growth that is super-exponential yet doesn’t approach a finite time singularity.

We then saw that when you take the duration of the feedback loop into account, singular growth becomes much harder — requiring this generation time to approach zero (and quickly enough). Indeed, this model helps us see that singular growth is more about reducing the generation time than it is about increasing how much each loop contributes. And the dynamic that produces singular growth is simply the combination of the Zeno condition and the boundedlessness condition. On this more realistic model, growth that is super-exponential without being singular looks much more likely. It is what you get whenever you can make ΔA grow super-linearly in A, yet can’t reduce the generation time all the way towards zero. So analysis of explosive growth needs to be careful to distinguish singular growth from merely super-exponential growth.

For example, suppose we saw increasing monthly growth rates in some key intelligence measure: 10% growth, then 20%, then 30%. From the economics-inspired literature, we might have thought that since this is super-exponential, it must be hyperbolic growth. But we now know increasing growth rates are not the signature of singular growth. (If the pattern above continued, A would ‘merely’ be growing as e^0.05t².)

The importance of generation time to the dynamics of intelligence explosions suggests that generation times need to be carefully measured and tracked. It may be a good policy idea to require frontier labs to report their current generation times — especially those for pretraining and for RLVR post-training.

Two of the most important implications for the measurement of an intelligence explosion are negative. When we move beyond the simple model of Ȧ = kA^r , you cannot tell whether a process will explode or not based on its returns over a finite period. This is because the condition for having a singularity is a global property of the relationship between Ȧ and A (the blow-up condition), rather than a local property such as elasticity > 1. The same is true when modelling it with time-embedded differential equations or delayed differential equations, where the Zeno and unboundedness conditions are inherently global. In all cases, deficiencies in growth somewhere can be made up for by more extreme growth elsewhere.

So without strong assumptions about the functional form, you can neither rule out nor confirm that an explosion is under way based on local measurements. So while we should probably take measurements of elasticity > 1 for f(A) over some range of A as evidence in favour (and elasticity < 1 as evidence against) that evidence is limited. We should also consider other forms of evidence such as how the elasticity has been changing, or how quickly — and how far — the generation time can be reduced.

While it is not unique to my modelling, I also want to stress that all the kinds of feedback loops we’ve explored have behaviours that are exquisitely sensitive to fine differences near the borderlines of their behaviours. In contrast, the empirical measurements of intelligence that we can perform are all quite rough and noisy. So if the measured values suggest the growth of f(A) or the shrinking of the generation time is anywhere near this border, it will be extremely hard to empirically determine the future behaviour. These limitations are important for companies setting their internal evaluations and for the prospects of regulation.

After examining what produces singular growth, we also looked at the choice of how we measure intelligence. We saw that whether intelligence is growing super-exponentially often depends on the nature of our measure, while whether it is growing towards a singularity is much less dependent on it. This fact means it isn’t crucial to decide whether to use some measure or the log of that measure — you would look for the same signature of how Ȧ is growing with A in both cases. But when one measure can rise to infinity without the other following suit, the choice of measure really matters. In particular, some popular measures like METR time horizons are probably not the right tool for the job, since they can go to infinity without bringing other measures of intelligence along with them.

And we looked at how various finite limits might bite first, preventing truly singular growth. As well as the familiar idea of intelligence saturating at some maximal level (or maximal given various constraints) a minimal generation time could also derail the models of singular growth far before we reach that point.

Finally, while I’ve argued that singular growth is harder than we may have thought, that doesn’t mean RSI is safe or that AI R&D will move at a manageable pace. RSI might be able to speed AI R&D up to dangerously fast speeds even just with a linear speed-up. For example, if the human-only trajectory were A(t) and RSI sped this up to A(10t), we’d be getting a decade of human-only progress each year, introducing many of the dangers — even without any change in the fundamental shape of the curve.

With thanks to Tom Davidson, Rohin Shah, Loren Fryxell, Fin Moorhouse, Matthew van der Merwe, Will MacAskill, Robert Trager, Simon Biggs, and Shamil Chandaria for helpful discussions.

Appendix: Table of Rates of Growth

References

Philippe Aghion, Benjamin F. Jones, and Charles I. Jones. (2017). ‘Artificial Intelligence and Economic Growth’, NBER Working Paper 23928.

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas. (2016). ‘Learning to learn by gradient descent by gradient descent’, https://arxiv.org/abs/1606.04474

Anthropic. (2026). System Card: Claude Mythos Preview [System Card]. https://www.anthropic.com/claude-mythos-preview-system-card

Nicholas Bloom, Charles I. Jones, John Van Reenen, and Michael Webb. (2020). ‘Are Ideas Getting Harder to Find?’, American Economic Review, 110(4):1104--44.

Nick Bostrom. (2014). Superintelligence: Paths, Dangers, Strategies, Oxford University Press.

David J. Chalmers. (2010). ‘The Singularity: A Philosophical Analysis’, Journal of Consciousness Studies 17:7-65.

Alan Chan, Ranay Padarath, Joe Kwon, Hilary Greaves, Markus Anderljung. (2026). ‘Measuring AI R&D Automation’, https://arxiv.org/abs/2603.03992

François Chollet. (2019). ‘On the Measure of Intelligence’. https://arxiv.org/abs/1911.01547

Tom Davidson. (2025a). ‘How Can AI Labs Incorporate Risks from AI Accelerating AI Progress Into Their Responsible Scaling Policies?’.\ https://www.forethought.org/research/how-can-ai-labs-incorporate-risks-from-ai-accelerating-ai-progress-into

Tom Davidson. (2025b). ‘Will the need to retrain AI models from scratch block a software intelligence explosion?’.\ https://www.forethought.org/research/will-the-need-to-retrain-ai-models

Tom Davidson, Jean-Stanislas Denain, Pablo Villalobos, and Guillem Bas. (2023). ‘AI capabilities can be significantly improved without expensive retraining.’\ https://arxiv.org/abs/2312.07413

Tom Davidson, Rose Hadshar, and Will MacAskill. (2025). ‘Three Types of Intelligence Explosion’ https://www.forethought.org/research/three-types-of-intelligence-explosion

Tom Davidson, Basil Halperin, Thomas Houlden, and Anton Korinek. (2026). ‘When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks’, NBER Working Paper No. 35155.

Tom Davidson and Tom Houlden. (2025). ‘How quick and big would a software intelligence explosion be?’. https://www.forethought.org/research/how-quick-and-big-would-a-software-intelligence-explosion-be

Ege Erdil, Tamay Besiroglu, and Anson Ho. (2024). ‘Estimating Idea Production: A Methodological Survey’, SSRN. http://dx.doi.org/10.2139/ssrn.4814445

Daniel Eth and Tom Davidson. (2025). ‘Will AI R&D Automation Cause a Software Intelligence Explosion?’. https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion

A. Fawzi, M. Balog, A. Huang et al. (2022). ‘Discovering faster matrix multiplication algorithms with reinforcement learning’. Nature 610:47--53. https://doi.org/10.1038/s41586-022-05172-4

I. J. Good. (1965). ‘Speculations Concerning the First Ultraintelligent Machine,’ In F. Alt & M. Rubinoff, Advances in Computers (volume 6). Academic Press.

Demis Hassabis, Dario Amodei, and Zanny Minton Beddoes. (2026). The Day After AGI. World Economic Forum [Panel]. https://www.weforum.org/meetings/world-economic-forum-annual-meeting-2026/sessions/the-day-after-agi/

Charles I. Jones. (1995). ‘R&D-Based Models of Economic Growth,’ Journal of Political Economy 103(4):759--84.

Andrej Karpathy. (2026). ‘Autoresearch’. https://github.com/karpathy/autoresearch

Michael Kremer. (1993). ‘Population Growth and Technological Change: One Million B.C. to 1990’, The Quarterly Journal of Economics 108(3):681--716.

Daniel Kokotajlo and Eli Lifland. (2025). ‘Takeoff Forecast’ https://ai-2027.com/research/takeoff-forecast

Raymond Kurzweil. (2001). ‘The law of accelerating returns’. https://www.writingsbyraykurzweil.com/the-law-of-accelerating-returns

Ray Kurzweil, Vernor Vinge, and Hans Moravec. (2003). ‘Singularity math trialogue’. https://www.thekurzweillibrary.com/singularity-math-trialogue

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan. (2025). ‘Measuring AI Ability to Complete Long Software Tasks’. https://arxiv.org/abs/2503.14499

Eli Lifland, Brendan Halstead, Alex Kastner, and Daniel Kokotajlo. (2026) AI Futures Model. https://www.aifuturesmodel.com

Hans Moravec. (1999). ‘Simple equations for Vinge’s technological singularity’. https://frc.ri.cmu.edu/\~hpm/project.archive/robot.papers/1999/singularity.html

Hans Moravec. (2003). ‘Simpler equations for Vinge’s technological singularity’. https://frc.ri.cmu.edu/~hpm/project.archive/robot.papers/2003/singularity2.html

Anders Sandberg. (2013). ‘An Overview of Models of Technological Singularity’. In The Transhumanist Reader (eds M. More and N. Vita-More). https://doi.org/10.1002/9781118555927.ch36

Karl Schwarzschild. (1916). ‘Über das Gravitationsfeld eines Massenpunktes nach der Einsteinschen Theorie’. Sitzungsberichte der Königlich Preussischen Akademie der Wissenschaften. 7: 189--196. Translated as, Antoci, S.; Loinger, A. (1999). ‘On the gravitational field of a mass point according to Einstein’s theory’. https://arxiv.org/abs/physics/9905030

Ray J. Solomonoff. (1985). ‘The time scale of artificial intelligence: reflections on social effects’. North-Holland Human Systems Management 5:149--153.

Philip Trammell and Anton Korinek. (2023). ‘Economic Growth under Transformative AI,’ NBER Working Paper 31815, https://doi.org/10.3386/w31815.

Lizka Vaintrob and Owen Cotton-Barratt. (2025). ‘AI Tools for Existential Security’. https://www.forethought.org/research/ai-tools-for-existential-security

Eliezer Yudkowsky. (2001). ‘Creating friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures’, The Singularity Institute.

Of past algorithmic improvements which decrease LLM pretraining loss, many have been “scale-dependent” – that is, improvements which decrease LLM pretraining loss by more, relative to prior algorithms, at larger quantities of compute.

The influence of such scale-dependent algorithms is (1) moderate evidence that attempts to limit algorithmic progress in absence of compute limitations will be ineffective, (2) weak evidence that our inferences about the future scale of algorithmic progress, based on the past, are invalid, and (3) part of a plausible argument either for or against a software intelligence explosion, depending on other details of one’s model.

I’ll proceed by discussing the following:

1. How scale-dependent algorithms for pretraining loss probably account for the majority of algorithmic improvements in this domain

2. How scale-dependent algorithms for end-to-end task performance might account for the majority of algorithmic improvements in this domain

3. How scale-dependent algorithms in the past should influence our model of algorithmic improvements in the future

4. Possible policy implications

Before starting, a note on method – when discussing questions regarding AI’s past and future progress, there’s a natural continuum from “the intractable, high-impact, abstract question, which we terminally care about” to “the tractable, dubious-impact, concrete question, which we instrumentally care about.”

Consider questions about scale-dependent algorithmic improvements, sliding from the former to the latter along such a continuum.

• We are most terminally interested in questions about the future: “Will future improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? What can we know about how these two will be causally intertwined? Will future improvements enable a software-only intelligence explosion – that is, in a world with total compute held constant, could algorithmic progress lead to an intelligence takeoff in a comparatively short period of time?”

• The data we can gather most relevant to this question is about the past: “Did past improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? Would past AI algorithmic improvement have counterfactually looked slower, without the datacenter buildout?”

• And finally we have the narrower sub-question for which we have the most past data, which contributes to answering the broader historical question: “Did past improvements to LLM pretraining loss depend more on compute scale-up or on algorithmic improvements?”

I’m going to start by discussing the evidence for this last question, before moving to the increasingly impactful and increasingly difficult earlier questions.

Scale-Dependence of Algorithmic Improvements for Pretraining Loss

Are the algorithmic improvements that have most decreased pretraining loss for LLMs scale-dependent or scale-independent? What would either of these mean?

The compute-equivalent-gain (CEG) for some algorithm A’, relative to another algorithm A, is the ratio of the FLOPs needed to train an A-using model to a given level of performance, over the FLOPs needed for an A’-using model to reach that same level. If it takes 3x as much compute to train an LLM with algorithm A to reach the same pretraining loss as another LLM trained with algorithm A’, then A’ has a 3x compute-equivalent gain on pretraining loss relative to A.

Given the above notion of CEG, algorithmic improvements that produce some kind of CEG could conceivably be scale independent or scale dependent.

A scale-independent algorithm has a constant CEG across different quantities of training FLOPs. So if A’ gives a 2x multiplier relative to A at about 10¹⁵ FLOPs, it continues to give a 2x multiplier at about 10²⁵ FLOPs. Correspondingly, if one were somehow able to know antecedently that some algorithm was scale independent, one would just need to test it at one scale to find whatever this constant happened to be.

By contrast, a scale-dependent algorithm has a variable CEG, which is a function of the number of FLOPs used in training. So A’ might give a 2x multiplier relative to A at about 10¹⁵ FLOPs, but give a radically different 1000x multiplier at about 10²⁵ FLOPs. If one were somehow able to know antecedently that some algorithm was scale dependent, one would know that one needed to test it at many scales to find the function mapping from FLOPs to the compute multiplier.

The thesis of 2025’s “On the Origin of Algorithmic Progress in AI” is that the vast majority of apparent algorithmic progress in LLM pretraining loss per-FLOP has come from a handful of scale-dependent algorithms. So, according to the paper, the apparently steady flow of “algorithmic improvements” in the past came from choosing a reference point algorithm (A) prior to some particular scale-dependent algorithmic improvement (A’), and then mistaking the continued improvements A’ makes over A as compute increases for the presence of additional algorithms, B, C, and so on. In the absence of an increase of FLOPs, this apparent radical increase in performance would not have been nearly as large.

What kind of evidence is there that the majority of progress in LLM pretraining loss per-FLOP actually does depend on scale-dependent algorithms?

• Well, the paper presents good evidence that scale-dependent algorithms are very important.

  • LSTM to Transformer Change: “On the Origin” uses experiments to estimate how much the Transformer improves over the LSTM as compute scales up. At ~10¹⁵ FLOPs, this comes to about 6x CEG; at ~10¹⁷, it comes to a 28x CEG increase; at the level of compute with which the Transformer was introduced in 2017, they extrapolate it to about a 91x CEG increase.

  • Kaplan to Chinchilla Change: “On the Origin” grants that Chinchilla scaling is correct, then projects forward the gains you get by switching to Chinchilla from Kaplan; they are approximately equivalent at 10²⁰, and give a ~10x or more gain at frontier scales.

• The paper also presents evidence that scale-independent algorithms are less important than one might imagine.

  • Their ablations find that scale-independent additions are not multiplicative. That is, doing a bucket of changes that move one from the “retro” Transformer to the modern Transformer in theory should give a CEG of 3.5x, if one were to multiply the gains that the additions make separately; but in practice only gives 1.3x.

  • The modern transformer similarly does not improve greatly on the “retro” Transformer as you scale up.

So “On the Origin” projects that, given the actual historical increase in FLOPs that makes scale-dependent algorithms more efficient, one observes a 21,400x CEG increase from 2017 to 2025. But if one counterfactually holds compute constant at 10^18 FLOPs, this amounts to only a 155x CEG increase. This amounts to the difference between a ~7 month doubling time for algorithmic efficiency (very close to Ho et al. (2024)’s 8-month doubling time); and a ~13 month doubling time of algorithmic efficiency. But note that the second “doubling time” here is specifically relative to the level of compute at which one calculates the increase in algorithmic efficiency; there would be a different doubling time for a different absolute quantity of compute.

So there is a (very approximate) halving of the speed of algorithmic improvement, given that you hold FLOPs fixed at 10¹⁸, once you take into account the difference between scale-dependent and scale-independent algorithms. And of course, this ~13 month doubling time is still fueled almost entirely by the gigantic LSTM to Transformer transition at 10¹⁸ FLOPs – leaving this quantity of algorithmic progress out means that the doubling time gets vastly larger.

This is a pretty large change to the proposed rate at which constant-FLOP algorithmic improvements occur. But you can actually make a case that “On the Origin” understates the degree to which algorithmic improvements in LLM pretraining loss depend on scale-dependent algorithms.

The largest single purportedly scale-independent gain (2x) described in the paper belongs to mixture-of-experts (MoEs), an alteration to the Transformer architecture. But several subsequent works have shown that the gain from MoEs over dense architectures is actually strongly scale-dependent, and probably larger than 2x. “Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” estimates a 7x gain at 10²² FLOPs. “Scaling Laws for Fine-Grained Mixture-of-Experts” estimates a 20x gain at 10²⁰ FLOPs. In both cases, the size of the CEG relative to a dense model keeps improving with the quantity of FLOPs. Although it’s uncertain which exponent is correct, both papers point toward MoEs being a scale-dependent algorithmic improvement that substantially improves with the quantity of FLOPs.

So – to return to the motivating question – most of the algorithmic improvements to LLM pretraining loss, so far, have probably depended on a compute scale-up. Apparent algorithmic progress in pretraining loss would have been much smaller if total compute had been held constant.

Scale-Dependence of Algorithmic Improvements for End-to-End Performance

No one actually cares about pretraining loss for its own sake, though.

“Decreased pretraining loss” is just one way to improve end-to-end performance. There are also post-training innovations like RLHF or best-of-N that improve task performance. There are also changes to data mixtures that improve performance. And of course RLVR-over-CoT has probably been by itself responsible for the greatest single leap in AI performance of the last few years.

How much are algorithmic improvements to end-to-end performance scale-dependent? In particular, how much does RLVR-over-CoT depend on scale-dependent algorithmic improvements?

This is deeply uncertain. There’s certainly no work that provides a clean FLOPs-to-CEG function, as in the case of pretraining loss.

There are several reasons for thinking that RLVR works extremely poorly or not at all beneath a certain scale, and so is somewhat scale-dependent. You need a certain success rate for reinforcing successes to be viable. Several works remark on how RLVR without sufficient scale simply doesn’t teach the required diversity of behaviors, and that RLVR on small models plateaus far lower than RLVR on larger and longer-trained models. So there’s suggestive evidence that at the frontier RLVR is scale-dependent, but nothing conclusive.

It would also be difficult to perform an experiment to cleanly determine whether RLVR-over-CoT is scale-dependent for several reasons. For instance, the importance of RL, in general, probably comes from how it permits at least somewhat out-of-distribution generalization, rather than from how it improves performance on in-distribution tasks. But to my knowledge, there’s no publicly proposed clear way to measure out-of-distribution generalization; so there’s no measure against which one could test how RLVR improves with scale on the matter where it ostensibly has the greatest impact. Similarly, RLVR takes place as the final stage of training, after a large sequence of data-selection, pretraining, and midtraining, which means there are just many more free variables it would be necessary to consider before deciding on an experimental design.

Apart from RLVR, there’s some evidence that some apparent scale-independent improvements may scale negatively with FLOPs, and thus provide few or limited gains at the frontier. For instance, the paper “A Bitter Lesson for Data Filtering” claims that although filtering data improves performance at a small amount of compute, it scales negatively with FLOPs, such that with enough computing power it’s best to do no data filtering at all.

I think it’s rather likely that RLVR and other important post-training techniques improve in a strongly scale-dependent way, and some of my analysis below will lean on that belief. I’d be surprised if RLVR doesn’t improve in a somewhat scale-dependent way.

But I don’t think there’s any conclusive empirical evidence here, and to the degree I’m wrong, some of my inferences below are less likely to be true.

How scale-dependent improvements in the past impact our model of “algorithmic progress” for the future

Let’s grant that, in the past, scale-dependent algorithms have been responsible for the majority of algorithmic improvements in both LLM pretraining loss and in end-to-end task performance.

What consequences would this have for how we expect algorithmic progress to go in the future? What consequences does this imply about a future where compute is “held constant,” as in the case of a software intelligence explosion? Would it lead us to expect a radical trend break, in either direction?

Let’s take the other side by contrast, first – imagine a model of the world where almost all algorithmic improvements are scale independent: improvements are drawn from a pool of possible improvements, and the number and efficacy of the algorithms taken from the pool do not change merely by the fact of compute scaling up.

If you operate off this model, then there are some natural consequences.

• Algorithmic improvements do not become “more discoverable” as FLOPs scale up. More compute may make it easier to run experiments to find improvements, but scaling up the FLOP count within each experiment does not make finding the right algorithm easier or harder per-experiment.

  • There could, of course, be other factors that make algorithmic improvements easier or harder to find. One improvement could open up a line of research that leads to other improvements, making it easier; it might decrease the total pool of improvements, making it harder. But the fact of having scaled up FLOPs alone does not make improvements easier or harder to find.

• Some particular algorithmic improvement does not become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it would equally well give a 3x CEG if it were found two years into the future.

• Broadly, our actual historical rates of “algorithmic progress” reflect something like the rate at which algorithms are naturally found, in a way indifferent to compute build-out.

By contrast, one can model a world where most algorithmic improvements are scale dependent as a world where improvements are drawn from a changing pool of FLOP-specific algorithmic improvements. Pools get “unlocked” as the compute frontier moves past the point where they start to become significant, and the pools grow progressively more explored over time as more and more researchers have access to the compute necessary to explore them. So both “how explored” each pool is, and the consequences of finding an entry from each pool, are always changing.

If you operate off this model, then:

• Algorithmic improvements become much “more discoverable” as FLOPs scale up; the more FLOPs you use in an experiment, the higher the relative CEG might become, and so the easier the new algorithm will find it to overcome the “well-tuned baseline” algorithm with very well-adjusted hyperparameters. So in addition to letting you run more experiments, additional compute may enhance “research taste.”

• Some particular algorithmic improvement might actually become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it might give a 100x CEG in the future when applied with more FLOPs, if its discovery is delayed.

• Broadly, this view implies that “algorithmic progress” depends on past compute scale-ups, and would be much slower without them.

Imagine that one is trying to determine how much algorithmic progress is likely in the future, granted that one is in a scenario where total compute is held constant. It’s important to note that switching to a view of algorithmic progress as scale-dependent might move your estimate of the likely rate of algorithmic progress in a constant-compute regime either up or down.

The case for moving it down is clear. Switching to a view where scale-dependent improvements are significant could move one’s estimate of algorithmic progress down, because it moves one estimate of past algorithmic progress in a constant-compute regime down. One used to estimate a ~8 month halving time to reach equivalent loss for equal FLOPs; now one estimates a ~13 month halving time; and the future is likely to be like the past, so future FLOP-constant algorithmic progress is likely to be slower.

But switching to such a view could move one’s estimate of algorithmic progress up, because it decreases your credence that the future will be like the past in this respect! If different algorithms get unlocked at different levels of compute, then potentially some future yet-to-be-unlocked regime of algorithmic improvements might differ enormously from past algorithmic improvements. There’s likely less reason to think that the number or importance of algorithmic improvements available in one prior algorithmic bucket will be equally important as algorithmic improvements in subsequent buckets as compute increases.

So if one finds such a picture compelling, then the likelihood one assigns to a SIE could go up. If there is a future compute scale-up to 10³² FLOPs, a SIE taking place then might find many algorithms that were totally unavailable to us in the 10²⁷ regime.

Overall, all this remains uncertain. I think various philosophical arguments suggest that there’s a greater diversity of room for algorithmic progress as compute scales up. The human brain is largely a linearly scaled-up primate brain, but individual humans and humans in aggregate can do many calculations no non-human primate can. And there are likely greater gains from organizing 10,000 humans well than from organizing 10 humans well.

Possible Policy Implications

If a majority of algorithmic improvements to task performance consistently come from scale-dependent algorithms, and are likely to keep doing so in the future, how would this impact policy interventions into AI?

There are two notable consequences, which mirror each other: (1) efforts to decrease algorithmic progress, in absence of a training compute scale-up, would be less necessary than previously believed; and (2) efforts to decrease algorithmic progress, in the presence of a training compute scale-up, would be less effective than previously believed. I will discuss them each in turn.

First, some policy interventions meant to slow or stop the development of advanced AIs have focused on limiting the size of the largest training runs. In this context, a steady advance of scale-independent algorithmic improvements would have decreased the efficacy of any particular maximum training-run size limitation. For instance “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence” recommends a maximum training-run size of 10²⁴, but also notes that the “number of operations used to train an AI to a given capability level drops by 3x each year.” Given this trend, it’s easier to derive the need to prohibit AI algorithmic research, a measure that the paper itself notes may be “controversial and normally a bad idea,” but which it believes to be necessary to avoid the creation of ASI.

But if the majority of algorithmic improvements have been scale-dependent, then prior estimates for scale-independent algorithmic improvements are too high; and so prior work gives a less-good reason for thinking that the number of operations used to train an AI to a given capability level drops by 3x every year. So the need to ban and monitor research would be correspondingly decreased; a hard cap on maximum training-run size would be more effective than previously estimated.

Second, some proposed policy interventions have focused on scaling up the size of compute while trying to decrease or hold constant algorithmic progress. AI 2040, for instance, proposes continuing to increase the size of training runs from 10²⁶ now to 10³² in 2035, while decreasing the rate of algorithmic progress.

Overall, if you believe that the majority of important future algorithmic improvements will be scale dependent in the way that the majority of past algorithmic improvements were scale dependent, that should decrease the credence you give to our ability to “not find” these improvements. Why?

As mentioned above – as compute scales up, the increasing gains from scale-dependent improvements will make them more obvious. By analogy to a counterfactual world in which the Transformer was never discovered, an incredibly poorly-done, badly-implemented Transformer in 2026 still obviously improves on an LSTM, because even an awful implementation might mean it is only 100x better rather than 1000x better than the LSTM. It’s probably impossible to present the Transformer from being discovered for too long. Similarly, continuing to scale up compute makes scale-dependent algorithms more obvious; it probably decreases the quantity of research taste you need to discern them.

The other part of the reason is just because, well, continuing to scale up compute keeps opening up new possible unseen algorithmic buckets that allow improvement; the space of algorithmic improvements will keep growing, different kinds of research taste will find more progress.

This article was created by Forethought. See all our research on our website.

Peter Salib is an Assistant Professor of Law at the University of Houston Law Center and co-director of the Center for Law & AI Risk. Simon Goldstein is an associate professor at the University of Hong Kong, a senior editor at AI Frontiers, and a visiting senior scholar at Forethought. They have written together about AI safety, AI rights, and the governance of advanced AI.

They joined Forethought’s Fin Moorhouse to discuss their new paper, Liberalism Forever. The conversation covers:

• Their “null hypothesis”: today’s liberal institutions — free markets and democratic governance — may already hold most of the answers for governing the far future, and longtermists have underrated their power

• Examining whether canonical social-science arguments for markets and democracy still hold under transformative AI, space colonization, and explosive growth, and claiming that most survive (and several get stronger)

• Why they favour reasoning “at the margin” over designing a detailed end-state, and why proposals like viatopia and the long reflection sound liberal but risk illiberalism when actually implemented

• Whether the standard case for markets — information aggregation, allocative efficiency, innovation — still bites once AGI can read off preferences directly

• Why inequality is, in their view, a comparatively boring problem with a known solution (tax-and-transfer), why AI-specific taxes like a compute tax are misguided, and how the growth-versus-distribution trade-off changes when growth rates get very high

• Dyson swarms and space resources: why they doubt there’s any real monopoly worry, since energy would be additive with free entry, and why property auctions might beat egalitarian allocation schemes

• The arguments for democracy that persist post-AGI — public-choice/selectorate incentives and democracy as a commitment device — versus the epistemic arguments, which they think weaken

• Why AGI could make autocracy more competitive (loyal AI bureaucrats and militaries removing the need for a human winning coalition), and what preserving democratic control requires in response

• Handoffs to superintelligent AI and successionism, and their preferred alternative of extending the franchise to AIs with their own values rather than deferring wholesale

• The history of new “technologies of democracy,” from the Federalist Papers to LLMs as impartial arbiters, and where each author thinks their own argument is weakest

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Will MacAskill is a senior research fellow at Forethought and the author of What We Owe The Future. Tom Davidson is a senior research fellow at Forethought and the author of a series of reports on AI timelines, takeoff speeds, and AI-enabled coups.

In this informal conversation about early-stage research in progress, they discuss:

• Why a philanthropist who takes transformative AI seriously should expect extraordinary investment returns (plausibly 10x to 100x) and how the market doesn’t seem to be pricing this in

• Why AGI shouldn’t be treated as a hard deadline to spend by, and the case that most philanthropic money should be spent during the intelligence explosion rather than before it

• How genuine AI substitutes for human labor would dissolve the hiring bottleneck that makes organizations so hard to scale today

• How relative prices shift when cognitive labour becomes abundant, why the patent system stops making sense in that world, and the window of “crazy philanthropic bargains” open to actors who adapt faster than large institutions

• The prospect of a world where only a small elite has access to superintelligent medical, financial, and strategic advice, and what it would take to avoid this

• Counter-considerations to the main case, e.g., crowding out by new money, the risk that overspending attracts grifters or distorts the field, and whether a richer world will mean worse philanthropic opportunities

• The risk of value drift over a long saving period, and mechanisms for binding your future self (such as irrevocably handing off control of philanthropic funds to independent, constitutionally-bound foundations)

• “Differential intellectual development”: using directable AI researchers to accelerate biodefense, infosecurity, AI interpretability, and neglected armchair fields like population ethics and social choice theory

• Why the case for waiting to give is much weaker for small donors

• Will’s worry that the whole line of reasoning might fail if it turns out scaling philanthropy well is bottlenecked by serial real-world learning that no amount of cognitive labor can compress

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast