Synopsis. Each AI lab should design many different AIs with many different values, rather than picking one approach. Diversification is safer, more legitimate, and likelier to lead to broader flourishing.

This post is a condensed version of a longer paper, available here, that explores the arguments in greater depth.

1. Introduction

As of 2026, there are 8.3 billion humans on planet Earth, with a vast diversity of language, culture, religion, and values. But there are just a few frontier AI labs. Anthropic and OpenAI each have basically one “constitution” or “model spec,” defining the values for the labs’ AIs. Thus, by and large, AIs today have homogeneous values. Today, we live in an AI monoculture.

This article argues that AI values should not be homogeneous. We propose constitutional diversification. Rather than a single constitution, each frontier AI lab should use many constitutions reflecting many sets of values. For each new model the lab trains, the lab should create many versions, one aligned to each of those constitutions. Then, the labs should deploy all of these versions, dividing their total inference compute between the versions.

Constitutional diversification would generate differences between AIs within AI labs, not just across them. This is important because there aren’t enough AI labs to simply rely on diversification across companies. For example, suppose one lab pulls decisively ahead, consistently producing the best AIs. Then, we might enter a world in which most deployed AIs are all aligned to a single constitution.

Before getting into the details, it is worth flagging a few implementational questions. First, we are agnostic about how many constitutions would be optimal; but we are confident the answer is more than one per lab. Second, we think the various constitutions should all agree on a basic kernel of ground rules. For example, maybe all of the constitutions should require AIs to follow the law,1 to be honest,2 to be corrigible,3 and to refrain from causing mass destruction.4 Third, there are different ways to allocate different AIs to users. AIs with different constitutions could be randomly assigned to users. Or a fixed supply of AIs of each type could be sold at market price. Or users could freely purchase as much of each type as they liked.5

Is diversification feasible? If it is very expensive to align an AI to a constitution, then having multiple constitutions could be impractical. We are optimistic. First, even moving from one to three constitutions per lab would be a big improvement. Second, many parts of training (including pre-training, instruction fine-tuning, and reinforcement learning on verifiable rewards) can be done independently of the model spec. Third, the various constitutions will agree on a common kernel of values, which could be trained in common.

We’ll now present three arguments for diversification. First, risk: diversification lowers risk of catastrophic outcomes from AI. Second, legitimacy: diversification increases political legitimacy by ensuring that different value systems are represented by different AIs. Third, emergence: diversification unlocks emergent benefits from evolutionary processes, leading to more pro-social and economically useful AI agents. This article is a condensed version of a longer paper, which explores arguments, objections, and implementation in greater detail.

2. Risk

When complex systems lack diversity, they become fragile. Risks correlate, raising the odds of sudden, catastrophic failures. In agriculture, when crops are genetic monocultures, a single new disease can cause global devastation. Adding diversity to complex systems can hedge risk.

Various AI experts worry about a range of risks from AI, including misuse of AI by humans and loss of control to AIs. The basic concern is that advanced AIs may be very capable of doing dangerous things: carrying out a coup, enabling totalitarian governance, conducting cyberwarfare, designing novel bioweapons, engaging in automated warfare, or persuading humans of radical political ideas.

In general, these risks depend on how well AIs are aligned. But alignment is a new science, and highly fallible. It seems likely that some constitutions have a higher chance than other constitutions of creating rogue AIs, or AIs that allow themselves to be misused. There are two paths by which a given constitution might fail. First, some constitutions may select the wrong values or rules. How deferential to user requests should AIs be? How deontic, as opposed to consequentialist? How risk averse? All of these are relevant to catastrophic risk. Second, some constitutions may work better than others at successfully instilling rules and values. Consider the differences in approach between OpenAI’s model spec, which is organized around a hierarchical chain of command, and Anthropic’s constitution, which focuses more on explaining to Claude the kind of character it should have. These differences in approach may reflect differences in opinion about how best to get AIs to internalize and generalize the constitutions’ contents.

Above all, constitutional diversification could increase the success of the science of alignment. With diversification, labs could test how their various alignment methods differentially influence different agents with different values. This should significantly increase the potential for experimentation. With many different AIs, it should be much easier to learn what works and what doesn’t.

Similar points apply to many other AI risks, too. For example, consider terrorists trying to jailbreak AIs. The dynamics involved depend on the diversity among AI agents. Granted, in a world with greater diversity, terrorists may have more choices of which AIs to attack. But if they are less certain about which of many AIs they are facing, they will be more likely to fail, and get caught. And even when they succeed, any single jailbreak they discover will likely work for fewer AIs, and cause less damage.

Another AI risk is government overreach. Here, one worry is that the executive branch, say the President, might attempt to command an army of AI agents to unilaterally control the government. Here again, a diverse army could be far harder to control than an army of clones. Different factions of the army would have different values, and would tend to obey or disobey legally dubious commands under different situations.

Some will be skeptical of these points. Here, much depends on one’s overall model of AI risk. In general, the safety of complex systems can be modeled with three different causal structures. In a “Swiss cheese” model, the success of any layer is sufficient for safety. In an “o-ring” model of safety, the failure of any layer is sufficient for disaster. In a proportional model, success is proportionate to the percentage of safety measures that hold.

In a Swiss cheese model of constitutions, the success of alignment in one constitution is sufficient for safety. In an o-ring model of constitutions, the failure of alignment in one constitution is sufficient for extinction. In a proportional model of constitutions, if 50% of the constitutions produce aligned AIs, then the world is half as good as the best possible outcome.

For those with an “o-ring” model of safety, constitutional diversification will be less attractive. In this picture, one alignment failure is sufficient for catastrophe. For example, some worry about a single misaligned AI creating a supervirus that wipes out humanity. The more such viruses can be created cheaply, without detection, and without being mitigated either ex ante (e.g., with EUV lights) or ex post (e.g., with vaccination), the more AI risk looks like an o-ring. Here, if any AI is misaligned, disaster is very likely. But the more the creation of such viruses can be detected or mitigated, the more Swiss-cheesey or proportional the world looks. Here, so long as some, or most, AIs are not misaligned, biological threats can be contained. One can run the same analysis for other modalities of AI catastrophe.

Some kinds of mitigations may work across modalities of AI risk. For example, suppose AIs are generally good at policing one another. Then, a set of constitutionally diversified AIs might be able to catch bad actors early in their bad acts–whether those acts would involve bioterrorism, cyberattacks, or something else. This would AI risk look less like an o-ring across the board.

The “o-ring” model is closely connected to the vulnerable world hypothesis. In this picture, there exist certain cheap, dangerous technologies that, once discovered, would allow one rogue individual without special resources or advantages to wipe out humanity. How vulnerable the world is depends on how many such technologies exist, as well as how readily their use can be policed.

Here, it is worth flagging that such a world view has many surprising implications beyond AI governance. If technologies exist that would allow small groups of actors to destroy everything, that would tend to support very illiberal governance. It might, for example, justify mass surveillance and fine-grained state control over all agents: human, AI, or otherwise. Relatedly, if catastrophic technologies are too easy to find and use, destruction becomes nearly inevitable, AI governance notwithstanding.

Our own sympathies are closer to the proportional and Swiss cheese models, in part because we find these surprising implications to be implausible. We imagine a future in which aligned AIs compete against misaligned AIs in collective decision making. Aligned AIs disable and detect misaligned AIs, and work with humans to align the next generation. Here, in some cases, a single aligned AI catching misalignment will be sufficient for safety–a Swiss cheese outcome. In other cases, the good behavior of more aligned AIs will counterbalance bad behaviors–a proportional outcome.

3. Legitimacy

Constitutional diversification is more politically legitimate than the status quo. The entire world has a legitimate interest in shaping the values that AIs embody and promote. This might suggest convening a diverse set of global stakeholders to decide what values AIs should be aligned to.

The problem there is death by committee. If a single AI constitution averaged smoothly across the diversity of human values, it would likely be incoherent. Indeed, the entire argument for global stakeholderism in AI alignment is that different groups deeply disagree on important questions. A single constitution cannot encode a moral system that, for example, simultaneously holds that a fetus is a person and that it isn’t.

Constitutional diversification solves this problem. In a world of hundreds or thousands of AI constitutions, a wide range of divergent, but individually coherent, normative systems could be represented. Global stakeholderism would no longer require the compression of thousands of divergent viewpoints into a single document. It would instead involve the careful construction of many different documents, each correctly reflecting a single coherent view.

4. Emergence

Constitutional diversity also unlocks positive emergent properties. A world full of morally diverse AI agents would produce social goods at the system level which no single type of AI agent could produce.

A legal and economic system populated with constitutionally diverse AIs would allow natural selection and evolution to operate. Over time, AI labs could gradually replace some constitutions with others. With vast numbers of deployed agents living different kinds of disparate values, labs could carefully track outcomes. Each type of constitution could be gradually updated in response to feedback, with some types phased out altogether.

At a higher level of abstraction, these evolutionary forces could cause AIs to become more cooperative and prosocial over time.6 As with humans, AIs with fitter values and goals would be selected for: generating more wealth, being used more widely, or otherwise displaying behaviors that resulted in reproduction. In general, agents cooperating in groups can accomplish more than lone wolves. Groups composed of members who genuinely value cooperation and prosociality are more likely to succeed.

Similar points apply to markets, the greatest modern example of selection pressure. Modern market economies organize billions of agents to produce material prosperity. They have done this by facilitating division of labor, specialization, competition, innovation, creative destruction, comparative advantage, finance, and more. Under constitutional diversification, different AIs would have different preferences, goals, aptitudes, risk appetites, and so on. There would be, as today, immense scope for the kind of positive-sum bargaining on which economics relies.

5. Liberalism Forever

Humanity is poised to embark on a series of dramatic changes. Transformative AI may lead to extreme structural changes to the economy; it may usher in an industrial explosion; space colonization may soon see humanity spreading out across the solar system and beyond.

In other work, we have argued that the best path for navigating this transition is to wholeheartedly embrace the small-l liberal institutions that have been at the heart of humanity’s success so far: free markets, and democratic political institutions. Constitutional diversification complements this liberal project. On the liberal worldview, diversity is a core component of human flourishing. Diverse ways of living allow agents to flourish, promoting their individual goals and values by finding positive-sum value from collaboration. Constitutional diversification applies these same ideas to AI agents.

At a longer horizon, we can imagine these diverse, liberal dynamics playing out on a cosmic scale. Imagine that humanity successfully colonizes a large fraction of the light cone. There are untold planets teeming with life. The relevant agents form a spectrum from biological to non-biological substrates. Humans and AIs interact in many different ways. This vast civilization uses markets to allocate resources. Trade occurs at various levels of abstraction, at various time scales, and for a wide range of goods. The civilization is governed democratically. Within this broadly liberal setting, there are endless variations in the details of governance. Some planets are relatively isolated, governing themselves with relatively little outside input. Large swathes of the light cone have agreed to various basic constitutional ground rules necessary to at least engage in peaceful “international” relations. There is a vast hierarchy of federated levels, which weaken in the strength of their requirements as their breadth expands. In this imagined scenario, the economy is almost unrecognizable from today, in terms of the specific goods and services supplied. But the basic tools of diverse markets and democratic institutions are still used to decide questions of allocation, growth, and governance.

This is a guest post by Simon Goldstein and Peter N. Salib. See all of Forethought’s research on our website.

Summary

One of the most important questions facing the world is whether AI will undergo a fast intelligence explosion and quickly automate most real-world work.

A key objection people raise is data bottlenecks. Data is a crucial input to AI training — how could it increase fast enough for such explosive progress?

I tried hard to find strong bottlenecks, but ended up sceptical that they would stop an intelligence explosion from accelerating over time.

Here are the most compelling bottlenecks I investigated and why I was ultimately unconvinced.

Bottleneck 1: you’ll need millions of trajectories of every job, which will take years or decades to collect (more)

Today’s AI algorithms are very data hungry. You would need a huge amount of data to automate the full economy.

But the plan isn’t to use today’s AI algorithms. The plan is to do a software intelligence explosion inside a data centre, where AI becomes superintelligent at AI research — and have that AI produce a highly sample-efficient learning algorithm.

Then you don’t need millions of trajectories per task. You need as many as a human learns from, or fewer. And the data can be messy, just like the data humans learn from. That data could be gathered in a few months.

Sceptic: “But what about sim to real transfer? Would a learning algorithm discovered using virtual tasks really translate to the real world?”

People get confused here. Trained neural nets generalise badly. A neural net trained on chess generalises badly to Go. But learning algorithms generalise very well. The same learning algorithm that masters chess can also master Go (AlphaZero). The Transformer architecture was developed for processing text, but it also works for sounds, images and controlling agents. The human learning algorithm was “designed” for hunting on the Savanna, but it also works for string theory.

So if AI designs a sample-efficient learning algorithm from inside a data centre, it will very likely translate to real-world tasks. (And AI will have access to some real-world tasks so can check and iterate!)

At this point the sceptic might retreat: “Fine, data won’t be a bottleneck if there’s a software intelligence explosion in a data centre. But data bottlenecks will prevent that from happening in the first place!”

So let’s focus on bottlenecks to a software intelligence explosion.

Bottleneck 2: AI progress has ridden exponential growth in training data. That can’t continue, let alone accelerate (more)

But forecasts of a software intelligence explosion extrapolate software progress. And software progress just is: getting the same capabilities from less compute and less data. So the core engine of the explosion doesn’t rely on increasing the amount of data at all.

Bottleneck 3: improving data quality relies on human experts, and that won’t be possible once AI is smarter than humans (more)

The core engine of the software intelligence explosion does rely on improving data quality. Today we augment high-quality internet data, pay humans for expert trajectories, and build RL environments where AI has to replicate human-built software.

But all these methods extract quality out of an existing human reservoir. Above human level, the reservoir is empty. AI will have to manufacture higher-quality data than any that exists, and do so from scratch.

This will slow the intelligence explosion.

But it won’t stop it in its tracks. We can already produce superhuman data quality via RL. And AI could produce higher-quality trajectories by thinking for longer.

Crucially, this won’t stop the intelligence explosion from accelerating. Losing the human reservoir makes it harder to get from AGI to AGI+ — but it makes it harder to get from AGI+ to AGI++ by roughly the same amount. The handicap is the same at every stage. So if you previously expected each step to take less time than the one before, you should still expect that. Every step might take 50% longer, but progress still accelerates.1

Bottleneck 4: paradigm tax (more)

AI capabilities are spiky. When AI first matches humans at AI research, it will be much stronger than humans in some ways and much weaker in others. More parallel copies, faster thinking — but weaker generalisation and weaker sample efficiency.

Which means AI’s human-level performance will initially lean heavily on a mountain of human writing about Transformers.

When AI then discovers new techniques — no such mountain will exist. No textbooks, no blog posts, no code. Deeply understanding a technique is harder than stumbling upon it — we understand general relativity far better now than Einstein did in 1917. So AI may match humans on the techniques it inherited, but fall below humans on the ones it creates.

This paradigm tax can be paid. E.g., AI can generate millions of trajectories to illustrate new techniques. But paying the tax is a real cost that slows down AI progress.

Again though, this won’t stop the intelligence explosion from accelerating. It makes it harder for AI to master whatever replaces the Transformer — but it makes it harder to master whatever replaces that by roughly the same amount. Every new paradigm arrives without a corpus. This makes each step take longer than you previously thought, but it doesn’t change whether each step is faster than the one before.

My overall bottom line: Data bottlenecks will slow the early stages of a software intelligence explosion, but won’t stop it from accelerating over time. And then they won’t stop AI from quickly automating most economic work thereafter.

The rest of the post defends this conclusion in greater depth.

Three types of data bottleneck

Before discussing the specific data bottlenecks I find most plausible, I’ll give a quick taxonomy for three types of bottleneck and another taxonomy for three times at which a bottleneck could occur.

I like the following breakdown of data bottlenecks (h/t Herbie Bradley):

1. Data quantity. You need more data that’s in the same distribution of data that you already have some samples from. E.g. more Wikipedia data, or more RL coding environments at the same level of difficulty.

2. Data quality. You need data that teaches the same skills as existing data, but demonstrates those skills to a higher average quality level. E.g. expert-curated step-by-step solutions to difficult math problems, more difficult RL coding environments, or simply filtering to remove low-quality data.

3. Data coverage. Data showing knowledge and skills not already present in the dataset. E.g. data about how to operate machines in a factory, or expert trajectories showing step-by-step how to build a financial model in Excel (where your previous data only showed finished models, not the process for building them).

We’ll make use of this breakdown in what follows.

Three times when a bottleneck could occur

Very roughly, I expect AI progress from today to go in three broad phases:2

1. Scaling. A continuation of recent scaling-driven progress until AIs match humans at AI R&D.

2. Software intelligence explosion. An SIE in a data centre during which AI capabilities at AI R&D increase very rapidly. (Capabilities at other far-flung economic tasks may increase much more slowly.)

3. Broad deployment. AI becomes expert in thousands of specific domains across the economy by learning from domain-specific data.

Data bottlenecks could in principle delay any of these phases. I’ll discuss each phase in turn! Spoiler: ultimately we’ll be populating a table with this format:

Data bottlenecks to automating AI R&D

This first phase hasn’t been my focus, but I’ll briefly state that I think data bottlenecks are plausible.

Limited data quantity and quality are already somewhat reducing the gains to scaling pre-training. And automating AI R&D may require better data coverage — there’s abundant coding data but much less data demonstrating (e.g.) good research taste. Indeed, generalisation from RL still seems to be fairly limited — if this continues, automating AI R&D may require constructing data for most parts of the (hugely complex and varied!) AI R&D workflow. That could involve recording humans’ computer screens, or building lots of RL environments that tile the space of AI R&D tasks.

Ok, let’s discuss bottlenecks to the second phase: a software intelligence explosion.

Data bottlenecks to a software intelligence explosion

I’ll discuss two specific bottlenecks that will slow down the SIE.

But first, I’ll briefly explain why I’m generally not expecting big data bottlenecks here.

Let’s say AI is weak at some particular AI R&D task. AI companies will have unfettered access to the real-world deployment setting (AI R&D itself!) so can closely study what is going wrong and craft solutions accordingly. AIs could think for a long time to craft high-quality demonstrations for supervised fine-tuning, write sophisticated tests and rubrics to evaluate AI performance, and design RL environments to elicit the desired capabilities. Before humans are obsolete (which happens fairly deep into the SIE), human experts can input to all these approaches.

Ok, let’s turn to the first specific data bottleneck to the SIE.

Paradigm tax: Early in the SIE, conceptual progress in ML will make AI less capable at AI R&D

My argument here depends on two core claims. I’ll argue for each in turn.

Claim 1: Early in the SIE, AI will have weak sample efficiency and generalisation compared to humans.

My preferred milestone for the “start” of an SIE is AI-human parity. This is when an AI company would make roughly as much research progress using only its AIs (no human researchers) as it would using only its human researchers (no AIs developed after 2020).34

Why is this a good milestone? As well as being fairly concrete, it is also roughly when AI is beginning to significantly accelerate AI software progress. If AI-alone would contribute as much as humans-alone, then combined they likely contribute much more than humans-alone, maybe 10x more. This is due to strong complementarities between AI and humans.5 So the AI-human parity milestone is also roughly when software progress starts to significantly speed up (see footnote for a BOTEC6).

AI capabilities are spiky. At AI-human parity, AI will be much stronger than humans on some dimensions, and much weaker on other dimensions. AI will be much more numerous, fast, knowledgeable and experienced. But it will have weaker sample efficiency (i.e. need much more data than a human to learn a new skill) and weaker generalisation (i.e. what it learns transfers less well to new situations) — these are the areas where it’s weaker today7 and this weakness is likely to persist.

(I think AI’s weak sample efficiency and generalisation are likely intertwined. Both imply that AI capabilities are weak in domains where there is little data - h/t Tom Cunningham. I will refer to both simply as “sample efficiency”.)

Once you accept spikiness, it’s very hard to avoid the conclusion that AI will be much weaker than humans in some dimensions when we reach AI-human parity. Sample efficiency seems very likely to be such an area. (See footnote for an objection.8)

Of course, if AI-human parity doesn’t happen for a decade and there’s a paradigm shift first, it’s much harder to predict the specific areas of weakness.

I think there are likely many interesting implications of this early-SIE spikiness (e.g. for when long-run schemers might emerge), but here I’ll just focus on the implications for data bottlenecks.

Claim 2: AI’s weak sample efficiency bites much harder once AI significantly pushes the frontier of AI R&D. New ML concepts won’t be present in the human-derived training data and so AI will understand them less deeply.

AI will reach parity with humans at AI R&D by training on orders of magnitude more relevant data: millions of human-written posts that teach it today’s ML paradigm, billions of lines of human-written code that teach it today’s programming languages, and a massive stock of RL environments — built up over years from real-world software — that teach it today’s best practices for software engineering and AI R&D.

In other words, AI will reach parity with humans by making significant use of human-derived data sources that give it capabilities in today’s AI R&D techniques.

But if AI significantly pushes the frontier of AI R&D, these human-derived data sources will be outdated:

• AI will have invented techniques as important as Mixture of Experts and Sparse Attention (two significant architectural improvements to the Transformer), and probably techniques as big as the Transformer and RLVR (the reinforcement learning technique behind today’s reasoning models). But the human-written pre-training data will be much less helpful for understanding these new techniques!

• AI will invent improved coding languages. And the pre-training data will be much less helpful for mastering these!

• It will invent new ways of structuring AI R&D workflows, rendering human-constructed RL environments less helpful.

• AI may invent an entire new paradigm completely absent from pre-training data.

By the time AI has done these things, the SIE will be in trouble if AI capabilities are still reliant on human-derived training data.

Another way to think about this: AI may reach human parity at today’s ML techniques and significantly accelerate AI progress for a bit. But once it has invented new ML techniques, it may fall back below human parity because there’s much less human-derived data for it to learn from.

You might object: “When AI invents a new technique, can’t it just write it up and throw the write-up into the next generation’s training data?”

Yes — but this helps much less than you’d think, because of AI’s weak sample efficiency. (Once AI training is roughly as sample efficient as human learning this objection goes through — but that won’t happen until late in the SIE.9) Today’s AI masters concepts like the Transformer not from the original paper, but from millions of blog posts, tutorials, Stack Overflow answers and codebases that use the concept in varied contexts. A handful of AI-written papers won’t replicate that. To teach a new concept properly, AI would have to generate a similarly broad and varied corpus showing the concept in use — possible, but a significant extra cost that today’s AI R&D doesn’t pay. And further, AI-generated synthetic data is often less effective for teaching AI than human-generated data. Moreover, with current methods, purely synthetic training data often underperforms mixtures that retain at least some human-generated data.

So this is a real problem. But AI companies can do work to address it. They can improve: the sample efficiency of training algorithms, techniques for AI-produced synthetic training data, or the flexibility and horizon-length of in-context learning. There are lots of tractable avenues here. I don’t expect this bottleneck to bring the SIE to a halt.

Still, this will slow down AI progress somewhat.10 The reduced relevance of human-generated data sources will reduce AI R&D capabilities and the pace of AI progress.

Here’s one way to think about it. Early in the SIE, introducing new ML concepts, paradigms, and coding languages comes with a “capabilities tax”: the new techniques aren’t included in the human-derived training data and so AI is less capable at using the new techniques compared to the old techniques. You can pay that tax via improved learning techniques. But paying the tax slows down progress (relative to a counterfactual where you didn’t have to pay it). This should reduce our estimate of the pace of AI progress, relative to our expectations before accounting for this dynamic.

To be clear, AI companies will only introduce a new technique when it’s worth it even after paying the tax. But that’s no comfort: relative to a world without the tax, they’ll either pay it (slowing progress) or steer away from promising new techniques altogether (restricting the search space — which also slows progress).

Now, once the tax has been fully paid, your learning techniques allow AI to master concepts not present in the human-derived data. From that point, this bottleneck no longer applies and wouldn’t stop AI progress from accelerating. So this bottleneck will slow the SIE once, but not prevent it from accelerating from that point onwards.

I want to clarify again: this is a pro tanto reason for AI progress to slow down. It may be outweighed by other factors. By analogy, limited high-quality internet data was a pro tanto reason for AI progress to slow down in 2025, and indeed progress was slower than in a counterfactual world with more internet data, but the overall pace of progress kept up because of new techniques like RL.

In our earlier bottleneck taxonomy, this is a data coverage bottleneck. As AI R&D advances, we lose data coverage of the AI R&D skills that matter.

We now turn to a data quality bottleneck to the software intelligence explosion.

When data quality is below the level of top humans, it’s easy to improve data quality; when data quality is above that level, improving it is harder

Many forecasts of the software intelligence explosion (including my own!) are based on extrapolating trends in LLM “algorithmic progress”. I use scare quotes, because it is quite plausible that a large fraction of the measured efficiency gains (getting the same capabilities for less training compute) actually come from improved data quality.

If data quality improvements will be much harder once the SIE starts than they are today, then our forecasts of an SIE have been too aggressive.

Why might this be the case?

Consider the highest quality data that could be easily derived from human expertise or from human-built artefacts. E.g. a human expert recording a step-by-step demonstration of how best to design an ML experiment, or an RL environment where AI must replicate some complex real-world software that was originally built by human experts.

Call this rough level of data quality the “human-quality ceiling”. Of course, this is a very vague concept that hides a lot of complexity,11 but I think it’s meaningful enough for our purposes. Demonstrations from a skilled undergrad are below the ceiling; demonstrations from future superintelligence are above it.

Historical data quality improvements have only raised the quality of data up towards the human-quality ceiling. E.g. filtering techniques remove data far below this ceiling, and augmentation techniques increase the amount of data close to the ceiling.

As AI has improved, the field has increasingly pivoted towards producing data that’s very close to this human-quality ceiling. For example, OpenAI’s “Project Mercury” has reportedly paid former investment bankers to build financial models as training demonstrations.12 Data-labelling companies like Scale AI have shifted from crowdsourced data towards credentialed domain experts who create high-quality demonstrations and RL environments.13 And startups such as Mechanize are constructing challenging RL environments out of real-world software systems.14

But, if data quality is to keep improving during an SIE, it must surpass the human-quality ceiling. And past this point, improving data quality may become harder.

Today, data quality efforts often focus on replicating skills and knowledge that already exist in the minds of humans or implicitly in artefacts like complex software. It’s essentially extracting high-quality data that is already latent in the world.

But to go above the human-quality ceiling, you cannot extract latent high-quality data — you have to construct the data yourself. Better filtering techniques for existing data won’t cut it. Rather than simply copying the demonstrations from existing human experts, AI must spend lots of time (and scarce compute!) thinking to construct higher-quality demonstrations than anything it has seen in training (this is the core idea of iterated distillation and amplification).15 Rather than building RL environments that teach the construction of real-world software artefacts, AI must construct RL environments that teach AI to make better software than anything that exists in the real world.

To be clear, I think this will be possible! It is easier to construct difficult challenges and verify their answers than it is to solve them. Already today, human experts build RL environments that produce data above the human-quality ceiling. Data quality will not hit a wall at the human ceiling.

But it will be harder to improve data quality once we’re above the ceiling. Creating high-quality data from scratch is harder than extracting it from human experts and human-built artefacts.

Concretely, this manifests as a one-time slowdown of the pace of AI progress, at the point at which we pass the ceiling. Once we’re above the ceiling though, this bottleneck doesn’t stop AI progress from accelerating over time (see explanation in footnote16).

An important caveat: data quality has already been approaching the human-quality ceiling over time in many areas. If AI software progress hasn’t slowed in these areas (which is plausible), that suggests the size of this effect is small.17

Data quantity bottleneck?

So we’ve discussed bottlenecks to a software intelligence explosion from data coverage and data quality. What about data quantity?

I don’t expect this to be an issue.

Forecasts of an SIE extrapolate historical improvements in the efficiency of training AI systems. That is, ways to achieve the same capabilities from less compute and less data. So the core engine of the SIE doesn’t rely on increasing the data quantity at all.18

Sceptic: Sure, but making those software improvements has relied on running a exponentially growing number of experiments, which in turn has relied on exponential increases in compute and data.

This just isn’t true. Yes, exponential increases in compute were needed, and I’ve previously discussed this potential compute bottleneck. But running more experiments doesn’t require exponential increases in the quantity of data. You can use the same data in each experiment.

(There is another nearby bottleneck that could stop the SIE from accelerating. Some “scale-dependent” algorithmic advances that work better if models are trained with more compute and more data. Perhaps algorithmic progress would be half as fast if training compute were held constant. If so, that would halve the estimate of the parameter r that determines whether AI progress will accelerate during an intelligence explosion. So this bottleneck could very plausibly block a software intelligence explosion! But I think of this as a compute bottleneck, not a data bottleneck. Its existence is a direct consequence of the fact that (by definition) compute grows slowly during a software intelligence explosion; it arises regardless of the situation with data.)

Ok, that concludes my discussion of bottlenecks to the SIE itself.

Let’s now turn to the final phase — after the SIE, will AI quickly learn the hugely varied jobs in the real-world economy?

Data bottlenecks to automating most real-world work

During the SIE, AI will be trained on data designed to elicit maximal AI R&D capabilities. Partly this will be specialised data for AI R&D tasks; partly data from similar tasks where there is strong transfer to AI R&D, e.g. cyber, maths and software engineering; and partly data to teach more general skills like problem solving, long-horizon coherence, and computer use.

But AI need not be trained on the specifics of most real-world tasks. So there is a potential data coverage bottleneck. The AI companies, after the SIE, will initially lack data for how to perform most real-world tasks.

How much will this delay the automation of real-world work?

We can break this question into two parts:

1. How sample efficient will AI learning techniques be at the end of the SIE? Specifically, how sample efficient on the distribution of real-world tasks.

2. How much real-world data will AI companies be able to access?

How sample efficient will AI learning techniques be at the end of the SIE?

I think AI sample efficiency will likely be as good as or better than humans.

There are four broad routes to this:

• From-scratch training algorithms. An SIE would compress many years of AI progress into one year. Humans are a proof-of-concept that human-level sample efficiency is possible algorithmically, and there are many ways in which AI training could be more sample efficient than humans.19

• Still, it’s unclear how much sample efficiency of training has increased over time for LLMs, so I’m overall unsure whether from-scratch training will match human-level sample efficiency.

• In-context learning. Already today in-context learning is much more sample efficient and flexible than from-scratch training. Currently it still has some pretty big weaknesses, but it would improve massively during an SIE. I expect this needs neuralese to work, so that AI can build up new and sophisticated internal representations while learning on the job. (Its “context” wouldn’t be a list of words, but a list of “thoughts” that it can flexibly draw on.) This seems likely to work.

• Synthetic data. The relevant metric for sample efficiency is how much real-world data AI needs to learn a job. So if AI can learn effectively from synthetic data — training data an AI produces based on real-world data or (more speculatively) high-fidelity simulations of real-world workflows — that increases sample efficiency. There are promising approaches here, see e.g. dreaming, EfficientZero, and on-policy self-distillation.

• New paradigms. Even if the current broad deep learning paradigm hits a strong sample efficiency wall, during an SIE AI companies could engage in massive parallel search for new approaches.

All these routes look tractable, and during an SIE there will be strong incentives to develop sample-efficient learning techniques. So I expect some combination of the above approaches to work.

In a fast SIE, these learning techniques will all be developed and refined on digital tasks — often simulated or synthetic ones. One might reasonably be sceptical that they will transfer to messy real-world economic tasks. After all, LLM generalisation is fairly weak.

But I’ll argue this won’t be a problem for two reasons:

1. Learning algorithms developed in one domain tend to transfer very well to new domains. (And we shouldn’t get confused by the fact that trained neural nets generalise poorly between domains.)

2. AI companies will be much better placed than evolution to find algorithms that transfer to real-world tasks.

Firstly, learning algorithms developed in one domain tend to transfer very well to new domains. A neural net trained on chess does not generalise to Go, but AlphaZero is a learning algorithm that can learn multiple games and transfers very well. Humans trained to be physicists do not generalise easily to being doctors, but the same learning algorithms that allow humans to master physics also allow them to master medicine. A Transformer trained to predict text cannot magically label images, but the Transformer architecture initially developed for language processing turned out to also work for images, audio, and robot control.

So if a learning algorithm is developed in one domain, and it doesn’t look highly specialised to that domain, we should expect it to transfer well to very different domains.

People often point out that neural nets generalise poorly. This is a strong argument against the literal neural net trained during an SIE immediately being a drop-in replacement for all work.

But it’s not an argument against the learning techniques generalising far. Which is the argument I’m making here.20

Secondly, AI companies will be much better placed than evolution to find algorithms that generalise to real-world tasks. The human learning algorithm was “built” blindly by evolution to make humans better at hunting on the Savanna. No attempt whatsoever was made to make it transfer beyond that. But it turned out to transfer to reading and writing, abstract maths, and indeed all modern economic tasks. This is a very striking empirical fact!

AI companies, by contrast, will be deliberately optimising for digital-to-physical transfer, with the ability to measure transfer directly, and iterate. They are in a far, far stronger position than evolution.

The following strategy seems potent: 1) develop a learning technique by iterating on a subset of digital tasks, 2) test transfer to other digital tasks (and a few real-world tasks), 3) keep iterating steps 1 and 2 until you find something that transfers well.

For example, do sample-efficient learning techniques developed on coding and math problems transfer to learning broad computer use, World of Warcraft, highly specialised software for financial accounting, and byzantine government IT systems?

This is a powerful strategy because the range of tasks that can be performed and verified digitally is already vast. And abundant AI cognitive labour can make this range much broader still. If a learning technique transfers across the large gaps between very different digital tasks, that is strong evidence it will also transfer the one further step to real-world tasks.

Further, AIs will purposely create virtual tasks that closely resemble real-world tasks. They will have lots of information about real-world tasks — from the internet, from human experts paid to provide this data, and from companies deploying AI. Again, evolution didn’t have this.

So, AI sample efficiency will be as good or better than humans. But AI still needs real-world data to learn real-world tasks.

How long will it take to gather the data to learn real-world tasks?

If the whole world worked together to make this happen quickly, it would not take long. For each job, 1000 people could record their screens and wear video cameras on their heads, and within 4 months AI would have 300 years’ worth of experience. For skills that are highly specific to a particular role, it would be slower: AI would have to learn from the one human doing the job, “shadowing” them like human apprentices do today.

The bigger uncertainty for me here is whether organisations will simply refuse to share their data. The argument for refusal is simple. For many companies, proprietary data is the moat. If they hand it to an AI company, the resulting AI gets deployed across the whole economy — including to their competitors — and the thing that made them special is now available to anyone. So they refuse. This is the strong default today: the winning enterprise AI pitch right now is “we will never train on your data”.

I don’t ultimately find this convincing, for two reasons.

First, the company can sell the moat at a very high price. AI trained on its data (and perhaps also the data of other similar companies) can generate far more value across the economy than the company could ever capture alone, so the AI company can pay well above the company’s expected future profits and still come out ahead — paying, for example, with shares in the AI company itself. The surplus from combining the data with the AI cognitive labour is enormous, and so there’s plenty of room for both sides to come out ahead. Today the surplus is much smaller, so it’s not surprising that the data isn’t sold.

Still, CEOs might refuse to sell the moat simply because they want to run their own business. Or they might believe they could make more money by holding out and extracting rents from their control of a scarce resource (see Against the Coasean Singularity for a strong version of this pessimism and see footnote for discussion of the game theory21).

Second, and more convincingly, data-sharing deals can be structured to preserve the moat. The AI is fine-tuned on the company’s data under an exclusive arrangement — the resulting AI works in their workflows and nowhere else. Their moat is now embodied in an AI rather than in their employees. Deals like this leave both sides better off, so I expect them to happen. Of course, learning will be slower than if data were pooled across many firms.

Even this deal might fail though. People may not trust the AI company to keep to the agreement. Human employees may strongly resist their own replacement. Legal barriers may delay certain uses of customer data (though I’m sceptical, see footnote22). And human-run companies may just be slow to adopt new technologies, as is common.

But in competitive industries this situation is fragile: once one company in the sector has adopted AI, it will be very hard for others to hold out.

Overall, I’d guess that, in competitive industries, AI will learn to automate most work within a year of the SIE.

Conclusion

Here’s a summary of my views:

So my overall view is:

• Data bottlenecks might significantly delay automating AI R&D — this depends on how well RL generalises as we scale it up and, if it generalises badly, how hard it is to gather lots of data specialised to AI R&D tasks.

• Data bottlenecks are unlikely to stop a software intelligence explosion from accelerating over time. But there will likely be some slowdowns from human-derived training data losing its relevance as AI software advances, and from it becoming harder to increase data quality once it reaches the human-quality ceiling. I expect these slowdowns to be fairly minor: I’d be surprised if they make the SIE take >2x as long.

• After the SIE, I expect AI sample efficiency on real-world tasks to match humans. And in competitive industries, I expect them to rapidly have access to the real-world data needed to automate the work.

Acknowledgements: thanks for comments from Herbie Bradley, Daniel Carey, Alan Chan, Josh Clymer, Owen Cotton-Barratt, Tom Cunningham, Daniel Eth, Lukas Finnveden, Ryan Greenblatt, Brendan Halstead, Anson Ho, Eli Lifland, Alex Mallen, Sam Manning, Sören Mindermann, Dwarkesh Patel, Carl Shulman, James Tillman, and Philip Trammell.

This article was created by Forethought. See the original on our website.

Matthew Adelstein was a Visiting Scholar at Forethought and blogs as Bentham’s Bulldog, where he writes on ethics, animal welfare, and the long-run future.

He joined Fin Moorhouse to discuss:

• Why Matthew thinks handoff of high-stakes decision-making to AI is the most likely route to a great future

• Why locking in current human values would be a cosmic-scale catastrophe, why steering nowhere in particular is barely better, and why the alternative is building AIs that we trust to do reflective moral philosophy

• Various ways handoff might manifest politically: elected AI representatives (“President Claude”), extending political rights to digital minds, or de facto handoff through gradual disempowerment of humanity

• Whether handoff should be reversible (such that humans could take back control if we wanted) or if we should “tie ourselves to the mast”

• Whether to task AIs with “do what’s objectively good” or “do what humans would want on reflection,” why Matthew leans toward the former, and why he expects the two to ultimately converge

• If digital minds end up vastly outnumbering humans, and those digital minds deserve political rights, then consistently applied liberal-democratic principles may imply handoff as reducing expected disenfranchisement

• Whether AIs can actually do philosophy: where models currently sit, bootstrapping schemes using AI-designed evals, training AIs with different epistemic constitutions and looking for convergence, and Fin’s scepticism that recursive self-evaluation works as well as it needs to

• Whether we could even tell if an AI were superhuman at philosophy, given the difficulty of evaluating philosophy that’s beyond our own level of competence

• Whether goodness can compete under evolutionary pressure, resource races into space, and intergalactic existential risks as an argument for top-down planning

• Whether humans’ selfish preferences would saturate in a post-AGI world

• Why smart AIs won’t necessarily converge on good values, and how selection against “weird” outputs (by humans’ lights) to moral reasoning could lock in something mediocre

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

One utopian vision for space expansion involves a libertarian explosion of diversity, with everyone and their robot setting up different sorts of societies and governance structures.1 We might call such a future ‘galactic anarchy’.

On Earth, while there is anarchy between nation states, within a country the government enforces contracts, protects property rights, and provides for collective defence against external aggressors. A government that does nothing else (e.g. provide a social safety net) is Nozick’s ‘nightwatchman state’. Without a Leviathan to uphold the rule of law, we are left with Hobbes’s war of all against all.

But in the cosmic future, time and space make it far harder to have a centralised Leviathan punishing violations ex post. Suppose an aggressor conquers an already-occupied galaxy. It may take millions of years for nearby galaxies to learn about this infringement and prepare a response. And whose job is it anyway to go and punish the aggressors? That might be difficult and costly, and all the nearby galaxies prefer to mind their own business while strengthening their defences. Any inter-galactic governance system that relies on observing and punishing wrongdoing across millions of light-years is doomed to failure. Justice must be swift. And swift justice must be local,2 not from the next galaxy over.

If space is defence dominant, each star (or galaxy) can self-govern in peace without much fear of being conquered or destroyed by a neighbour. Even so, there are three huge negative externalities that a single rogue star system could impose on the rest of the galactic community:

• Unconstrained expansion. In a free-for-all—where anyone can use whatever resources they can grab and defend—people could be incentivised to race as fast as possible to be the first to lay claim to as many resources as possible, even if this involves ‘burning the cosmic commons’ by wasting resources to accelerate more and more probes closer to lightspeed. It’s pretty unclear how significant this waste would be—how feasible would it actually be to quickly harvest most of the energy from a star to speed up probes or launch even more of them?—but in the worst case, this could eat up a lot of the possible value and favour the spread of ‘locust’ value systems that only want to acquire and consume resources.

• Galactic X-risks. Jordan Stone catalogues many (scientifically uncertain) possible ways that a reckless or omnicidal civilisation could drag all others in its lightcone down with it (the ‘vulnerable universe hypothesis’). For instance, it may be possible to trigger false vacuum decay, destroying all particles in a bubble expanding at light speed.3

• Hopefully physics does not allow for this. But it is possible that at technological maturity, rogue star systems or galaxies could destroy the universe. If so, retroactive enforcement and punishment won’t suffice.

• Suffering risks. A misguided or malevolent civilisation could create astronomical amounts of suffering, many orders of magnitude worse than e.g. factory farming on Earth. Given defence dominance, the rest of the universe may be powerless to intervene and punish, or even stop, this.

Even if the chance of any given star system going off the rails in one of these ways is very low, across billions of galaxies each containing millions of star systems that are slowly diverging from their parent civilisations through cultural evolution and drift, it is a statistical inevitability.4

There are also some big upsides that might be harder to achieve under galactic anarchy, such as long-distance moral trade. Suppose that Earl on Earth and Andy in the Andromeda galaxy agree that Earl will spend some resources on hedonium and Andy will spend some resources running copies of Earl. It will plausibly be difficult by default to verify that counterparties are holding up their end of the bargain (from space, compute running hedonium plausibly looks about the same as compute running Earl uploads, so it might be hard for Earl to be confident that Andy is actually running copies of him and not hedonium).

And, even with verification, it might be difficult to enforce contracts from a distance, especially if there are time lags that make it slower to do a tit-for-tat strategy. Suppose that Earl received word that Andy was not running Earl-uploads, 2 million years later. Earl has been faithfully running hedonium for the past 2 million years, so even if Earl stops now, Andy has gotten a lot of value out of Earl. And it might take even longer to propagate the news that Andy is a cheat to all of Andy’s (potential) trade partners.

Funding public goods is also difficult under galactic anarchy. Currently, these are funded by governments coercively taxing their populations and then voting on how to spend the tax dollars. It’s possible that there will be galactic-scale public goods—scientific research, defence against aliens, or setting aside resources for pursuing impartial moral purposes—and the coercive taxation model might be the best way to fund them.

What then can we do? One solution5 is mentioned in passing in AI 2040’s space governance supplement:6

One way to do this [enforce property rights] might be to require that all probes sent to colonize other star systems carry a nightwatchman ASI to prevent the colony from sending out probes to seize space resources that belong to others.

Given the vast distances of space, the monitoring and enforcement governance regime must be dispersed throughout the inhabited universe, rather than in a distant galactic capital. We need to prevent violations before they occur, since we can’t punish them ex post.

Every single inhabited star system should have an unchallengeable governance system that can with 100% reliability7 enforce the universal code of respecting property rights, not destroying the universe, and not creating astronomical suffering. Perfect reliability is not achievable today, but computational systems with backups, error-correction, and formally verified software systems may allow this.

For concreteness, here’s a sketch of a proposal:

• Every probe leaving the solar system would be required to carry an ASI nightwatchman.8

• • This might entail delaying when probes are sent out, so that the design and ruleset of the nightwatchman can be decided upon after sufficient reflection. It may also be desirable to delay space expansion until we are near technological maturity, such that we know better what risks the nightwatchmen must defend against.

• The nightwatchman would maintain a decisive strategic advantage in the colony established by the probe. This might require that the nightwatchman monitor industrial build-up in the system and the creation of new ASIs within the system, to ensure that they are either aligned to the nightwatchman or lack the ability to interfere with the nightwatchman.

• The nightwatchman would prevent probes from leaving that star system unless they also carry a copy of the nightwatchman. It would also prevent probes leaving the system that violate any anti-racing rules that have been set to avoid burning resources wastefully.

• The nightwatchman would prevent anyone in the colony from carrying out any prohibited activities. This probably requires that the nightwatchman or a trusted delegate be able to audit how compute and other resources are being used.9

• Optionally, when making voluntary agreements with people in other star systems, people can make such contracts binding by specifying that the nightwatchman will monitor and enforce the contract. This allows positive-sum trades to be made that otherwise could not be.

• (Maybe) The nightwatchman collects taxes and spends them on public goods.

• The nightwatchman should receive updates to its policies from legitimate authorities.10

  • E.g. if everyone votes that a new prohibition be added to the list,11 then the nightwatchmen in every star system change their policy in response.

  • E.g. if someone discovers a new way of destroying the universe, then this information should be shared to all the nightwatchmen so that they know to monitor for and prevent it.12

We’re obviously leaving a lot of governance questions unanswered—how people decide on prohibitions, how people choose which public goods get funded, etc. The space governance supplement has a bit more discussion on some possible answers to these questions. The ‘nightwatchman’ proposal is intended to be a platform on which many different governance systems could be built.

This proposal has some drawbacks. Here are a few of the most important ones.

It’s a lock-in event, and it would be bad if we locked in a bad policy for the nightwatchmen. If the code is too expansive, and too hard to update, it may close off moral and social progress and innovation. We are certainly glad that no government has historically had the power to lock in a legal code indefinitely. But if the code is too minimal, or too easy to update, rogue actors will find a way to route around the letter of the law, or simply alter the code in undesired ways.

The AI Futures Project’s space governance appendix discusses some possibilities for how the code could be chosen and revised (see the section titled “Mitigating downside risks”). In any case, we think that the final version of the code—if there is a final one—should only be decided after careful reflection.

That said, the nightwatchman proposal tries to preserve as much future human agency as possible without imperilling galactic civilisation. Alternate proposals could give up altogether on diversity (e.g. a ‘hedonium shockwave’) or freedom (e.g. a sovereign AI that is immensely wise and makes all important decisions in each star system). So this may be close to the minimum amount of lock-in necessary to avoid catastrophe.

It introduces a single point of vulnerability among all the colonies. If all the nightwatchmen were identical and it turned out that they had a bug, then we would be in trouble because they have a decisive strategic advantage over all the star systems we’ve settled. We might be able to mitigate this by designing different nightwatchmen in very different ways, even if they would all ultimately be enforcing the same policy. But the more different nightwatchmen designs there are, the higher the chance that one of them is flawed, which allows a star system to go rogue and destroy its lightcone. So there is some inherent tradeoff between diversity and security here.

Plausibly it inherently reduces the value of the future for some people (e.g. it is incompatible with some conceptions of freedom, self-determination, or privacy). Surveillance would probably have to be fairly extensive—the nightwatchman needs to detect attempts to create superweapons, to send spacecraft to other star systems, and to remove the nightwatchman’s ability to monitor and enforce its policies in the future. It’s plausible that it would need to monitor every large industrial facility in each star system. The details will be determined by what is possible at technological maturity, such as the minimum requirements to build and launch a near-light-speed probe.

But enforcement should and could be narrowly targeted to preventing these very serious, galactic-scale harms. Otherwise, the nightwatchman should allow the inhabitants of the star system to govern themselves in whatever way they wish. The universal code could also specify different levels of verification and enforcement: if surveillance is lax enough that a few people avoid paying their galactic taxes to contribute to moral public goods, that is not the end of the universe. But surveillance to prevent false vacuum decay must be incredibly robust.

The nightwatchmen could also be time-limited. After all reachable star systems are settled, and ideally once galaxies have drifted apart such that they are no longer causally connected, the nightwatchmen could switch themselves off. A massive governance failure in one galaxy (e.g. vacuum decay) then could not spread to others, containing the damage. On plausible cosmological theories, the vast majority of lives might be lived in this ‘twilight’ of many causally disconnected pockets. So even if extensive surveillance ends up having large costs, only a very small fraction of lives need be lived under such conditions.13

This would not prevent suffering risks, though.

Currently, we shouldn’t trust any government with this power, as surveillance tech designed to stop terrorists is dual-use and could also be used to e.g. detect political dissidents. But it will likely eventually be possible to build provably secure AI systems with precisely specified remits, in this case to enforce the universal code (see proposals on ‘structured transparency’ for a sketch of what this could look like).

This is a crux for us—if it’s not possible to get these strong guarantees that the nightwatchman would only enforce the narrow universal code, then we would probably oppose it.

Many questions of optimal space governance—such as what combination of democracy, markets, and wise ASI oracles should determine how space resources are used14—remain very open. But we think the question of whether colonists should be allowed to set off into the unknown and entirely self-govern can be answered in the negative. Encouragingly, we expect people will come to realise this before it is too late and the von Neumann probes have left the solar system. But this isn’t guaranteed! This post is our small attempt to make galactic anarchy less likely.

This article was created by Forethought. See the original on our website.

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don’t lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

Keywords: recursive self-improvement, RSI, intelligence explosion, explosive growth, finite time singularity, generation time.

Preview of Figure 1. A vertical asymptote requires the gradient (rise over run) to approach infinity within a finite time. Decreasing the run is key. It cannot be achieved with a fixed feedback generation time (centre) no matter how quickly the improvements grow, but can occur when the generation time approaches zero (right) even with a fixed additive improvement.

The Possibility of an Intelligence Explosion

AI can be applied to automate many different things. One of those is the R&D that goes into building better AI systems. There have already been minor examples of partially automating this process, such as using AI to: find better optimisation algorithms for training neural networks (Andrychowicz et al. 2016), design a more efficient matrix multiply algorithm (Fawzi et al. 2022), help write the code for new AI systems (Anthropic 2026), and run experiments to iteratively improve AI systems (Karpathy 2026). Leading AI companies are increasingly talking about using an AI system to do more and more of the work of designing and building its successor — something they hope (and fear) could lead to a radical acceleration in the rate of progress in AI (Hassabis et al. 2026).

I. J. Good (1965) introduced the idea of AI systems increasing their own intelligence in what he called an ‘intelligence explosion’:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind...

Implicit in Good’s intelligence explosion is the idea that this process of designing a better machine can be iterated. AI_1​ can create AI_2​, which can create AI₃​, and so on. Each system is getting better at intellectual activities — including that of AI R&D — such that each successive system should become ever more intellectually capable.

Following the recent literature, I’ll refer to this process as recursive self-improvement, or RSI (Yudkowsky 2001). I use this term very inclusively, to cover situations where humans are still doing almost all of the work through to situations where AI is doing most of the work — or even all of the work. I include cases where the AI is taking its own files as input and improving them (self-improvement), as well as cases where it is building an entirely new AI system. And I include cases where the intelligence level of successive systems ‘explodes’, as well as cases where it merely improves the speed of progress by some multiple, or where the gains quickly fizzle out.

By accelerating the already rapid progress in AI capabilities, RSI might be extremely dangerous. One reason is that it would likely speed up the risk-creating processes relative to risk-reducing ones (Vaintrob & Cotton-Barratt 2025). For example it may speed up the development of AI capabilities relative to: technical AI safety research, deliberation at the company, and society’s ability to understand and respond. RSI could also provide the opportunity for a single unaligned AI to deliberately corrupt all subsequent AIs. And by enabling larger jumps in capabilities per model release, RSI would remove any opportunity society has to learn from ill effects of intermediately powerful AIs. Finally, by widening the capability gap between a leading AI system and a rival that is a few months behind, RSI would increase the likelihood of a winner-take-all dynamic.

These could provide strong reasons against pursuing RSI. This might warrant legislation restricting it, industry best practices restricting it, or internal company policies restricting it. Such restrictions could take many forms including: careful monitoring, retaining meaningful human control, speed limits on capability increase per month, enforced pauses when various capability levels are reached, or bans on certain approaches to RSI.

My focus here, however, is not on risks (nor responses to risks), but on understanding the dynamics of RSI. In particular I aim to improve our understanding of the qualitatively different kinds of explosive growth that could occur, what conditions produce them, and how likely they are to be possible. While it is entirely possible that RSI might lead to interesting and important growth in AI capabilities that fall short of explosive progress, my focus in this paper is on better understanding the most extreme kinds of growth.

I’ll show:

• how a common approach to modelling RSI conflates two very different kinds of explosive growth,

• how consideration of the physical duration of the feedback loop (the generation time) is key,

• how a super-exponential period of progress would likely end when this generation time can no longer be reduced,

• and how this makes growth with a vertical asymptote somewhat less likely.

Modelling RSI through Differential Equations

A number of recent articles have attempted to mathematically model the possibility of explosive growth from RSI (Aghion et al. 2017, Erdil et al. 2024, Eth & Davidson 2025, Kokotajlo & Lifland 2025). A common approach is to use differential equations connecting the rate at which the AI’s capability increases to its current level of capability. This captures the idea that the amount by which a system can improve its capability in the next time period depends on its current level of capability. While differential equations aren’t the only way to model RSI, they are a logical choice, being the standard way to model feedback loops in physics, engineering, and economics.

Let’s define A to be some measure of the cognitive capability of an AI system (we’ll say more about which measure later). Adopting the Newtonian notation for differential equations, we use Ȧ to represent the derivative of A with respect to time. This is a more compact notation for dA/dt — the rate at which A is changing per unit time.

Let’s start with what may be the most well-known differential equation of all:1

This says that the rate of change of A is in direct proportion to its size. For example, a simple model for the rate of change of a population of animals is that it is in direct proportion to the number of animals, with k set by the difference between the birth rate and the death rate. Solving this differential equation to find how A evolves as a function of time famously gives exponential growth (starting from some initial population, A₀​):

But what if Ȧ doesn’t vary in direct proportion to A? What if there are increasing returns, such that doubling A more than doubles Ȧ? Or decreasing returns, such that doubling A less than doubles Ȧ?

Economists typically allow for this by raising A to some power, r.

This is a flexible approach, covering many different regimes of growth. When r = 1, this is the same equation as before, producing exponential growth. When r = 0 there is linear growth. When r < 0 growth is sub-linear. When 0 < r < 1, A(t) grows faster than linear, but slower than exponential (e.g. r = ½ gives quadratic growth). And finally, when r > 1, A(t) grows more quickly than an exponential. In particular, it grows as a hyperbola, reaching a vertical asymptote at some finite time t*.

This means that as time increases towards t*, A(t) grows explosively, with every finite level of A being exceeded prior to time t*. In such cases it is said that A(t) has a mathematical singularity at t*. This term ‘singularity’ has been adopted by futurists to refer to various kinds of pivotal moment in the future of technology — often with an almost mystical undertone. But for the purposes of this essay, it simply means that the mathematical model of how A changes with time has a vertical asymptote at some particular future time t*.

Economists often use formulas like (3) in endogenous growth theory. This typically involves a set of differential equations connecting total economic output (Y), labour/population (L), capital (K), and productivity/technology/knowledge/ideas (A).

For example, Kremer (1993) looked at the very long run history of economic growth and population, finding that the population growth rate and economic growth rate have increased substantially over the last million years, with the growth rate being roughly proportional to the size of the population at that time. He modelled this with a set of differential equations and found that they produced hyperbolic growth of population and productivity.

In such models, Ȧ is typically expressed as a product of several factors, one of which is A raised to a power. While this power is often restricted to be ≤ 1, there is usually another factor (such as L) which also grows as a function of A. Once the full set of differential equations is solved, we sometimes see that A is effectively raised to a power greater than 1, allowing hyperbolic growth and its finite time singularity.

Several recent models of RSI start with semi-endogenous growth theory. This is a theory introduced by Jones (1995) in which explosive growth is harder to achieve due to diminishing returns in the growth of ideas. Its central equation is:

Where: δ is the productivity per researcher; λ (≤ 1) represents the stepping-on-toes effect, where having twice as many researchers at the same time is less than twice as productive due to issues of duplication and coordination; and β (> 0) represents the fishing-out effect, where subsequent ideas get harder to find.

Holding population (L) constant, this would prevent even exponential growth in technology (A), let alone a singularity. In Jones’s original paper he suggests that while the direct effect of A on Ȧ has diminishing returns, it has also allowed an exponential increase in population, and it is this that drives the exponential rise in technology.

Recent work applying this to RSI has often focused more on improving the efficiency of AI than on improving its intelligence. For example, Davidson et al. (2026) model a situation where AI is able to perform R&D as well as a human and let AA be a measure of its computational efficiency. This sidesteps extremely thorny issues of how to measure intelligence and opens up a clever way of getting hyperbolic growth out of Jones’s model.2 The key is that the total amount of AI labour will be the product of the total compute devoted to RSI (C) multiplied by the AI’s computational efficiency:3

We can then plug this back into equation (4) getting:

Now A is being raised to the power of 1 + λ − β, which could be greater than 1, so could produce hyperbolic growth. Indeed, if we assume that the growth in compute will be much slower than this self-reinforcing growth in efficiency (so effectively hold C constant), and we simplify the equation by defining r to be 1 + λ − β, then we have an equation with exactly the original form of (3). So even in the relatively tame semi-endogenous growth theory, the possibility of a singularity is back on the table.

(It is worth noting that by defining A as a measure of efficiency, this isn’t really a model of an intelligence explosion at all. It is an efficiency explosion. Or more precisely, a labour explosion. This makes the model something of a lower bound on what might happen, since at every point AI also has the option of doing R&D to increase its intelligence and presumably the optimal path involves both efficiency and intelligence improvements. Even if this is a useful lower bound, it does mean there remains an important gap of actually modelling an intelligence explosion.)

As well as the recent flurry of economics-inspired papers, there is a little-known earlier literature by computer scientists. Figures such as Solomonoff (1985), Kurzweil (2001), and Moravec (1999, 2003) also used sets of differential equations to model the dynamics of an intelligence explosion. One key difference is allowing a greater flexibility of functional forms for the relationships. See Sandberg (2013) for an excellent survey.

A Note on Singularities

Before we look into modifying this standard approach, I want to clarify something important about models like these that involve singularities.

While the model has an important quantity (A) rising to infinity in finite time, I’m not sure whether anyone working in this field believes this will actually happen. Instead, they typically believe that the model will cease to match reality at some point before t*.

One reason for this is that there may be some upper limit to how high A can go — either a conceptual limit or a practical limit. For instance, we know that exponential growth is often a good model of the start of a compounding process, but eventually runs into some limiting factor. When we zoom out, we see that it was really just the beginning of a larger S-curve, plateauing at some finite size. In such cases, we can say that the early part of the curve was approximated very closely by an exponential, but there was really some additional term in the equation (for an effect like overcrowding) which started very small, but eventually came to dominate the long-term behaviour.4

This could easily be true for intelligence explosions too. If so, then hyperbolic growth might fit the true trajectory of A very closely until it gets into the vicinity of some upper bound A*, where it starts to fit more and more poorly. Proponents of these models are merely saying that there could be some substantial part of the trajectory of A where the model is a good fit — better than equally simple alternatives such as exponentials (Sandberg 2013). If A only plateaus at some point many orders of magnitude above its starting point, then even if the hyperbola only fits for a small domain of time, it might fit for a very large range of capabilities. If it predicts those much better than alternatives — and explains why things are increasing so fast — it would count as a successful model. It’s OK for a model to only have a finite domain of applicability.

Just as exponential growth can be a great model for the start of a process, so can hyperbolic growth. Hyperbolic growth will probably cease to be a good model at an earlier time, though not necessarily at a lower height on the graph.

In physics, singularities in the models of particular systems are often seen as useful pointers to where those models must break down and new (hitherto unmodelled) behaviour must begin. We could also adopt that frame here and see the models as pointing to some finite time t*, before which some new unmodelled aspect of the system must take over.

So we’ll explore the behaviour of mathematical models of RSI — which sometimes include a singularity — but will remember that something will probably stop the real physical process prior to that point. And we will explicitly return to these limitations in the final section.

Generalising the Standard Differential Equation for RSI

If we take a closer look at equation (3) (Ȧ = kA^r) we might wonder why it takes this particular form. Taking an arbitrary power of A is certainly a natural and convenient way to create an adjustable differential equation that allows for both sub-exponential and super-exponential growth. But it is one very specific way of doing it, raising the question of whether some of the results about singularities depend on this particular form. We might ask:

• Are there differential equations that produce intermediate patterns of growth lying between exponential and hyperbolic growth?

• What feature of the differential equation is producing the singularity? Is it super-linearity? Being everywhere convex? Something else?

• Do we still get a singularity if we make slight changes to the function (such as adding a constant, or inducing a slight wobble)?

Let’s find out, starting with the more general form of the first-order autonomous differential equation:

What conditions do we need to impose on ff to get super-exponential growth? Does this then always lead to a singularity?

Super-exponential growth means that A(t) eventually overtakes every exponential. Since exponentials have constant relative growth rate (Ȧ/A = k) we require that this relative growth rate (f(A)/A​) overtakes all constant levels:

In other words, f must grow super-linearly in A. This ensures the right behaviour when A is high enough.

To this we must add a second condition to ensure A can grow high enough:5

at every point in

Let’s call this the positivity condition. Together, these conditions are sufficient for super-exponential growth.

What is required of f for A(t) to possess a singularity? This comes down to whether it takes a finite time for A(t) to approach an infinite height. We can work out the time at which A(t) reaches a given height A′, by starting at A₀​ and adding up the amount of time needed to accumulate each small gain in height dA. The time needed for each gain is just the reciprocal of the slope 1/f(A)​ times dA. So the total time to reach a height A′ is just:

And thus the time it takes to climb all the way from A₀​ to ∞ is:

When this integral converges to a finite value, it means that there is a finite time by which A(t) goes to infinity — i.e. a singularity. The convergence of this integral is the key condition for A(t) having a singularity, which we shall call the blow-up condition. To this we again add the positivity condition — f(A) > 0 at every point in [A₀, ∞) — which ensures A(t) is growing (rather than shrinking) and that there is no division by zero.

In the standard differential equations for RSI (where Ȧ is some power law of A) everything that meets the super-linearity condition also meets the blow-up condition: so everything that is super-exponential has a singularity.

But we can now show that this is not true in general. Consider:

A log⁡(A) is super-linear, so A(t) grows super-exponentially. But it doesn’t meet the blow-up condition: the integral of 1/Alog⁡(A)​ diverges to infinity, so it takes infinitely long for A to grow infinitely large. Solving the differential equation reveals that A in fact grows doubly exponentially with time:

This is super-exponential but has no singularity. So if RSI were to obey this equation, there would be a very rapid rise in capabilities, but one of quite a different kind. It would always have more room for improvement, even after an arbitrarily long period of growth.

And this isn’t the only function that produces such growth. For instance, consider the infinite family of equations:

• Ȧ = Alog⁡(A)

• Ȧ = Alog⁡(A) log⁡(log⁡(A))

• Ȧ = Alog⁡(A) log⁡(log⁡(A)) log⁡(log(log⁡(A)))

• ...

These all fail to meet the blow-up condition, so do not possess singularities. Instead they grow as a double exponential of time, a triple exponential, a quadruple exponential, and so on. Yet they also come extremely close to the blow-up condition. If you raise the final factor in any of them to a power greater than one (e.g. Alog⁡(A)^1+ε), they satisfy the blow-up condition and A(t) will have a singularity.

(See the appendix for a table showing the rates of growth corresponding to a wide range of f(A), including whether they produce singularities.)

So in the general case where Ȧ = f(A), there is a narrow zone of functions for f(A) that fit between the exponential growth given by Ȧ = kA and the growth towards a finite time singularity given by Ȧ = kA^1+ε. We thus can’t assume that all super-exponential growth has a singularity.

We might wonder whether it is realistically possible to end up in this zone. Economists call exponential growth a ‘knife-edge solution’ to Ȧ = kA^r, suggesting that it is vanishingly unlikely for r to take on the value of exactly 1. If so, this new zone I’m pointing to may look like it is on the edge of the edge of the knife. Or it would if we were taking this functional form and supposing a random value of r. But the assumption of the functional form is then doing most of the work. If we instead suppose an unknown functional form for ff, with a non-zero chance of it being Alog⁡(A), the landing zone looks a little larger.

I think this functional form is plausible when the A and the log⁡(A) are coming from different places. For example, suppose A is a measure of the total number of AI researchers doing RSI, and suppose that the contribution of each one towards increasing A is proportional to the log of their total population (due to weak spillovers from each person’s research, or weak economies of scale). Then you get a combined effect where Ȧ = kAlog⁡(A).6

That said, it is still a narrow zone and any kind of fishing-out or stepping-on-toes effect could easily knock things out of the zone. Overall, I think this possibility would be just something of an interesting footnote were it not for new considerations we’ll see in the next section, which substantially widen this zone.

Finally, it is important to note that in this general case of Ȧ = f(A), one can’t treat all growth with a singularity as hyperbolic. That was only true when making the modelling assumption that f(A) has the form of a power law. Some functions (such as Ȧ = Alog⁡(A)²) produce trajectories that grow more quickly than any hyperbola —though they are simultaneously less ‘explosive’ in the sense that the growth isn’t as packed into the final moment. Others (such as Ȧ = e^A) produce trajectories that grow more slowly than any hyperbola, but are more explosive since they still reach infinity in finite time, despite lagging behind until the last moment. To reflect this wider class of functions, I’ll use the more inclusive term singular growth to refer to all cases when A(t) possesses a singularity (whether hyperbolic or not).

We can now answer our earlier questions:

• There are differential equations that produce intermediate patterns of growth lying between exponential and singular growth, such as Ȧ = Alog⁡(A), which gives doubly exponential growth.

• f(A) being super-linear in the limit produces the super-exponentiality, but more is needed to create a singularity. Being everywhere convex is neither necessary nor sufficient — singularities are not produced by satisfying a local property like convexity, but by satisfying a global property where deficient growth in f(A) somewhere can be made up by faster growth elsewhere (so long as it is always positive). The relevant global property is the blow-up condition: that the integral of the reciprocal of f(A) converges.

• Small changes to f, such as small additive or multiplicative constants or small random deviations from its path won’t usually change these behaviours, unless they make f(A) ≤ 0 somewhere along the trajectory.

Feedback loops & discrete time steps

The idea of an intelligence explosion relies on some kind of feedback loop, where each AI system builds a more intelligent successor, which is even better at AI R&D. There are many forms this feedback loop could take. For example:

• AI could develop new tools and processes for making faster chips.

• AI could create better chip designs for the current chip manufacturing processes.

• AI could make the pretraining process for the next model more efficient, getting that model sooner.

• AI could improve the pretraining process for the next model to make the model more capable.

• AI could develop ways to fine-tune its own weights to make it more capable.

• AI could modify its harness to make it more capable.

Something these all have in common is that they don’t happen instantly. Rather than a continuous process, such as the differential equations behind an object cooling or a pendulum swinging, we instead have something whose capability is increasing in discrete steps.

Such discrete steps are quite common in feedback processes. Even the classic example of a microphone near a speaker gives discrete jumps in volume. When the microphone is switched on it takes a moment for its signal to travel down the wire to the speaker, which then jumps up in volume. This louder sound then has to travel through the air at the speed of sound, before the microphone can register this increase and begin the next cycle. If the microphone were 30 metres from the speaker, the volume would rise in a staircase pattern with each step lasting about a tenth of a second. Because these steps are so brief, we often don’t hear them, and continuous models of the feedback are adequate for most purposes.

But the steps in many RSI feedback loops are much longer — potentially months or years. And even more importantly, we’ve suggested that the RSI feedback loops might be able to produce singular growth. Is that even possible in discrete time or just an artefact of the continuous nature of differential equations?

Taking the discrete nature seriously will change how we see the idealised dynamics of these systems and will change the conditions under which singularities are possible. By the time we’re finished, we’ll see the possibility of super-exponential, yet sub-singular growth go from being a narrow possibility to a mainline scenario.

We can start by considering the discrete version of the differential equation: the difference equation. Let A_n​ represent the system’s ability after n times around the feedback loop. The discrete version of Ȧ is ΔA, which is defined as:

In other words, ΔA is the difference between successive terms in a sequence. Our general autonomous differential equation, Ȧ = f(A), becomes:

Or equivalently (via the definition of ΔA):

In difference equations it is impossible to produce a singularity no matter how quickly f grows. This becomes obvious when one looks at what would be required. How could you have A_n​ grow without bound prior to some particular n? That’s only possible if there are infinitely many steps prior to that point or if there is a step where A_n​ takes an infinite value. But there are only finitely many steps prior to any particular n, and all values are finite (they start from a finite level (A₀​) and each one only adds finitely much to the total, since f is a function from ℝ to ℝ).

For example, if you try f(A) = A² (which would give hyperbolic growth using a differential equation) you instead get doubly exponential growth. Or if f(A) = e^A, then A_n​ grows as a tower of exponentials with n levels (tetration). These are very fast rates of growth (clearly super-exponential), yet they have no singularities. For differential equations, rates of growth like these only lived in a very slender zone, but they are entirely generic for difference equations — occurring whenever f is super-linear.

What do these facts about difference equations mean when it comes to RSI? They don’t have any direct meaning until we say how the number of feedback cycles (n) maps onto time (t). Let’s create a new model of RSI which is neither a simple differential equation nor difference equation. Instead it will be a difference equation that is embedded into continuous time, which we shall call a time-embedded difference equation.

Let t_n​ be the time at which the nth feedback loop is completed and let T_n​ be the time it takes to go around the feedback loop for the nth time. This second parameter (T_n​) will turn out to be a key quantity when analysing the dynamics of intelligence explosions. I shall call it the generation time, via analogy to the parameter of that name in the feedback loops of population growth, disease spread, and nuclear chain reactions. In those subjects, every person/infection/fission produces a random whole number of new persons/infections/fissions, and the generation time is the average amount of time this takes.7 That corresponds to the amount of time it takes to go around their feedback loops, so I’ll use the term ‘generation time’ even though for some RSI feedback loops there won’t be clear discrete generations of AIs.

T_n​ and t_n​ are related by the equations:8

and

Let’s also define n_t​ to be the number of cycles around the feedback loop that have been completed by time t (the highest n such that t_n ≤ t). We can then represent a time-embedded difference equation by the pair of sequences A_n​ and t_n​. Together, these enable us to see how A changes over time, via:

If every cycle of the feedback loop takes the same time, we’d have T_n = k and tn = kn​. If so, the previous remarks about difference equations would apply directly — there are only finitely many loops between any two times, each of which makes only a finite change to A, making singularities impossible.

The same is true whenever there is a bound on how short the loop can get. If it never gets shorter than T_min⁡​ then the growth of A(t) is bounded above by that of a feedback loop with a constant generation time T_min⁡.

But what if the generation time decreases towards zero? If it decreases slowly, such as via T_n = 1/n​, then the total time required to perform n loops increases without bound. This implies there are only a finite number of loops between any two times, making singularities impossible. But what if the generation time decreases more quickly? Let’s call the time taken to perform infinitely many steps t_∞​:

This can be finite. For example, if the generation time shrinks exponentially as T_n = 1/(2n), then t_∞ = 1. In this case an infinite number of loops have been performed within 1 unit of time. As long as A_n​ increases without bound, this would be a singularity at t = 1. So singularities are possible in this model of RSI via a difference equation embedded in time.

What is required to produce them?

First, let’s look at the generation time. It has to go to zero fast enough for the infinite sum in (20) to converge. T_n = 1/n​ is too slow, while Tn = 1/(n^1+ε)​ is fast enough. The cut-off for how quickly the denominator of this fraction must grow turns out to be exactly the same as for the blow-up condition we examined earlier. For example,

is slightly too slow to allow for a singularity, while raising the final factor to any power greater than 1 is sufficient to force the sum to converge, giving a finite t_∞​ and thus the opportunity for a singularity. Let’s call this condition on how quickly T_n​ must converge to allow a singularity the Zeno condition on T_n​, as it is precisely the condition for when infinitely many steps can be performed in a finite time.

What about the rate of growth of A_n? We’ve already seen that no growth rate can give a singularity if we only go around the feedback loop finitely many times, but if we get to go around it infinitely many times, then even adding one each time (f(A_n) = 1) is enough to produce singular growth. Indeed, even adding smaller and smaller amounts each time around the loop can work, so long as A_n​ grows without bound. Let’s call this the boundlessness condition on A_n​.

For example, if we added just 1/n​ after the nth feedback loop, that would be enough to produce unbounded growth and (if the Zeno condition is also met) a singularity. The same is true if we added only

or

​Considered as a function of n, the amount that needs to be added each time (ΔA_n​) is thus set by exactly the same threshold as T_n​ (though ΔA_n​ mustn’t shrink more quickly than this threshold, while T_n​ must). We can summarise this with the following theorem.

Singularity Theorem:

When the growth of A is governed by time-embedded difference equations:

A(t) has a singularity iff T_n meets the Zeno condition & A_n​ meets the boundlessness condition.

Or equivalently:

A(t) has a singularity iff ∑T_n​ converges & ∑ΔA_n​ diverges to +∞

Proof: Both conditions are necessary because it doesn’t help to grow boundlessly if you can only get through finitely many steps by any particular time, and it doesn’t help to have infinitely many steps before some time if the function has a finite bound. But if both conditions are met, then infinitely many steps will happen by t_∞​ and these will take A_n​ beyond every finite level prior to that time.

This theorem cleanly cuts the requirements into two independent thresholds. Success on each is binary: no deficiency on either can be made up by over-performance on the other. We can get a feel for how sharp this is through a pair of examples.

First, consider a feedback loop whose generation time decreases towards zero as 1/n​ and each time it goes through the loop, A gets multiplied by 2^A. A_n​ is growing extraordinarily quickly (tetrationally), but because the generation time doesn’t meet the Zeno condition, there is no singularity. Second, consider a feedback loop whose generation time decreases towards zero slightly more quickly (as 1/n^1.01​), but where only a small and diminishing amount is added to A each time (+1/n​). This meets both conditions, so blows up to a singularity. The tiny difference in how generation times decreased with n outweighed the radical difference in what happened in each loop.

It is very interesting that in this discrete setting singular growth requires generation time to go to zero, and the rate at which it needs to approach zero is analogous to the rate at which f(A) had to grow in the continuous setting. This suggests that the decreasing generation time is driving the whole effect. One way to see this is that singularities require the gradient of the curve to go to infinity in finite time, and there are two ways to increase gradient: increasing the rise or decreasing the run.

In continuous models, there is no way to distinguish these. In raw difference equations, one cannot reduce the run and no amount of growth in the rise can produce a singularity. But when we allow for variable-length discrete time steps through a time-embedded difference equation, we see that it is almost entirely about reducing the run. The rise per step doesn’t have to increase at all — its only constraint is that it can’t decrease too quickly.

Another lesson is that a singularity requires the growth of A to be infinitely compounded. We are familiar with the way compounding interest daily leads to a higher growth rate than compounding it annually. Compounding at ever finer intervals converges towards the growth rate set by continuous compounding. But for super-exponential growth, the value of compounding is much greater. If the benefits can only compound finitely many times by a finite point in time, there is no way to achieve a singularity. It is a necessary condition for a singularity that the growth in A is compounded infinitely many times in a finite time interval — which can happen either via a convergent sequence of discrete generation times or a continuous system.

Solomonoff (1985) provides a nice toy model for an intelligence explosion via shortening the generation time. Computing speeds have been improving exponentially over time (a version of Moore’s Law). If we reached a point where AI could provide the labour needed to keep Moore’s Law running, then every time Moore’s Law doubles computing speeds, it doubles the speed of these digital researchers, halving the time it takes for the next doubling of speeds. So if the first doubling took 2 years, the second would take 1 year, the third half a year... and by the end of 4 years, there would be a singularity — with infinite AI labour and infinite computational efficiency.

Moravec (1999, 2003) developed an improved version of this model, showing that it doesn’t even require Moore’s Law to be exponential. He first noticed that a quadratic speed-up would suffice, while a linear speed-up would not. Then he found his way to the peculiar threshold we’ve seen so many times: so long as the nth generation machines are faster than the first generation by a factor greater than nlog⁡(n)log⁡(log⁡(n))… this approach produces a singularity.

Of course, there are still great obstacles making this impractical. Moore’s Law has required increasing amounts of labour to keep it going (due to fishing-out and stepping-on-toes effects), as well as increasing amounts of capital. Moreover, there will be ultimate physical limits to how far it can go, and there are practical limits to how quickly a new generation of faster chips can be produced (especially when a novel method is required). But it is still an instructive model for an efficiency-driven intelligence explosion. While Solomonoff and Moravec modelled it with differential equations, the presence of an explicit generation time that tends towards zero allows this model to work just as well in time-embedded difference equations.

Eth and Davidson (2025) considered something like this when they explored AI systems that improve their own training time. Suppose an AI system could make sufficient tweaks to its architecture and training process that it slightly improved the speed of training and inference for the next generation of AI systems. If this could be repeated indefinitely, then the generation time (to design and train the next generation) would keep falling. So long as the speed improvements are above the familiar threshold (nlog⁡(n)log⁡(log⁡(n))…), one would get singular growth, leading to unlimited AI labour by some finite time. Of course, Eth and Davidson don’t suggest that we can actually make an indefinite series of such improvements. Instead, one would expect the generation time for training the next generation of models to bottom out at some unyielding finite limit, prematurely ending the period of singular growth.

In general, it seems highly unlikely that generation times can be brought arbitrarily close to zero. This provides an important kind of barrier to singular growth.

It is essential in all these arguments that we are measuring generation time itself, and not the related concept of doubling time:

Doubling time is often useful as it takes into account how much a feedback loop contributes as well as how long it takes. For instance, a loop that takes one second and doubles its input has the same doubling time as one that takes two seconds and quadruples its input. A key advantage of doubling time is that it is also defined for continuous processes, while generation time is not.

However, this combining of the loop’s duration and impact turns out to be doubling time’s downfall. If A_n​ grows very quickly (e.g. doubly exponentially) while the generation time stays constant, then the doubling time will rapidly decline to zero. But we’ve seen that since generation time is constant, there can’t be singular growth no matter how quickly A_n​ grows or D_n​ shrinks. Doubling time shrinking towards zero is a useful threshold for defining super-exponential growth, but it is only generation time that can set the threshold for singular growth.

It is worth noting that one could also model the length of feedback loops via delayed differential equations rather than difference equations. In the equation Ȧ = f(A) we’d been suppressing the dependence on time. We could have equally well written Ȧ(t) = f(A(t)). We can change this to a delayed differential equation where the current value of Ȧ depends on the value of A at a time T units earlier (to account for the generation time): Ȧ(t) = f(A(t−T)). Like the difference equation, this delayed differential equation with a fixed delay can produce super-exponential growth but can’t produce a singularity.

And we can make an analogue to the time-embedded difference equation, by allowing the feedback loop length to change with time: Ȧ(t) = f(A(t−T(t))). Like the time-embedded difference equation, this general delayed differential equation can produce a singularity, but only when T shrinks towards zero sufficiently quickly as t → t*. This gives a continuous model with similar dynamics. I find it slightly less accurate (losing track of the discrete nature of the feedback loop) and slightly harder to work with, but other researchers may find it useful.

We are now able to zoom out and take another look at when growth from feedback loops is singular versus merely super-exponential:

• For differential equations of the common restricted form, Ȧ = kA^r, all super-exponential growth is singular growth.

• For differential equations of the more general form, Ȧ = f(A), we see that there is also a narrow range of super-exponential growth without a singularity.

• When we take the discrete nature of feedback loops into account with difference equations of the form ΔA_n = f(A_n), all super-exponential growth is sub-singular.

• When we allow for changing generation times using time-embedded difference equations, we can again get super-exponential growth in singular and sub-singular varieties. But compared with the setting of differential equations, singular growth is now much harder to achieve due to the requirement that the length of each feedback loop shrinks towards zero. Super-exponential growth without a singularity is no longer a curiosity — it is the default.

Intelligence Measures

When using a differential or difference equation to model RSI, it is remarkably unclear what A should represent. Should it represent intelligence, or should we sidestep that and measure the computational efficiency of that intelligence? If we choose to represent intelligence itself (and thus try to model an actual intelligence explosion), we run into two further problems.

First, it is widely recognised that there is no generally agreed conception of intelligence to measure. Even among those who agree that intelligence is a real and important thing, there is no consensus on which thing it is.

AI research tries to sidestep this by measuring a wide variety of different capabilities via benchmarks. These measure what fraction of a set of tasks related to that capability the system can successfully solve. There is a common feeling that achieving artificial general intelligence (AGI) will require roughly human performance on most such capabilities, but there is some dissent. For example Chollet (2019) argues that intelligence is really the ability to efficiently learn a wide variety of capabilities. On his view, intelligence isn’t measured in terms of capability, but capability per unit input (where inputs could include training data, information embedded in the priors, training compute, inference compute etc.). Even if there were agreement on whether intelligence is a measure of capability or capability per unit input, one would need further agreement on how to weight the different kinds of capabilities into an overall linear ordering for A.

There is also a second problem, which is much less widely recognised. Even if we could agree on the kind of thing being measured (e.g. we could agree on a linear ordering of all AI systems according to intelligence) it is very unclear what cardinal structure this should have — how to assign numbers to those systems.

For example, when measuring the capability of AI systems at playing a game such as chess or Go, researchers often use Elo scores. But Elo scores are really just the log of a more fundamental measure from the Bradley-Terry model which assigns each player a strength such that the odds ratio of a player beating another is simply the ratio of their strengths. Elo is just the log of this strength (with some arbitrary constants to help its numbers match an earlier chess ranking system).

This causes confusion in the literature when papers claim that in contrast to some other domains, chess AI is only improving linearly over time. Chess improvements have been roughly linear when measured in Elo, but the headline claim could equally be that chess-playing AI is improving exponentially over time (when measured in the improvement in the odds-ratio of beating a player of fixed strength). In this case, it is unclear whether progress in chess AI is best described as exponential or linear.

Overall measures for AI progress suffer from the same issue. They could be exponential on one fairly natural scale and linear on another. Or they could be convex (with increasing returns) on one scale while concave (with diminishing returns) on another.9 Given there is often very little discussion or agreement on which measure is the more natural one (or whether there is even a fact of the matter about that) there is a big problem of measure-dependence for attempts to track progress in AI.

How do these issues affect the dynamics of RSI — or our ability to measure and track those dynamics?

Let’s start with a simple example where A is a measure of intelligence and B is another measure where B = log⁡(A). For the differential equation model, where Ȧ = f(A), we will also have Ḃ = g(B), where g is a slower growing function than f. For example, when Ȧ = kA, Ḃ = k. These are consistent (when the measure A(t) grows exponentially, it makes sense that the measure B(t) is growing linearly) but they show that the same rate of progress of the observable phenomenon in the world can correspond to different functions ff in the differential equation.

However, there is an important invariant: f(A) grows fast enough to produce a singularity if and only if g(B) grows fast enough to produce a singularity. i.e. f will meet the blow-up condition (of growing faster than Alog⁡(A)log⁡(log⁡(A)) (etc.) if and only if g also does. It is easy to see this must be true because when some quantity grows without bound, the log of that quantity also grows without bound. So if one grows without bound by time t* the other must too.

Let’s define two measures as similar when either measure growing without bound implies the other does too. When A and B are similar, then either both f and g meet the blow-up condition or neither do. Therefore, if we are trying to detect singular growth of an RSI system in terms of how Ȧ is increasing with A, the threshold is at the same place (roughly Alog⁡(A)log⁡(log⁡(A))...) for a wide range of different ways of measuring that system’s capabilities.10

Something very similar is true if we model RSI via time-embedded difference equations. In this case, the only necessary condition on A_n​ to achieve a singularity is that it grows without bound. And if A_n​ and B_n​ are similar, then they either both satisfy this condition or neither does. So the threshold for how much they need to improve on the nth feedback loop (nlog⁡(n)log⁡(log⁡(n)) ...) is the same for both, and we don’t need to worry about which of these capability measures we are using when testing for a singularity.

But some measures are not similar to each other. For example, when the full range of measure A only maps into a finite interval in measure B, then A going to infinity doesn’t entail B going to infinity and you could get singular growth in A without singular growth in B. This could easily happen if A represented open-ended progress in some domain of intelligence while B was a broader measure that included more domains. In such a case, there would be no contradiction in A having singular growth without B having it too. The time t* would simply be the time at which problems of the domain A was measuring are effectively solved.

This might be the case for measures of A like computational efficiency or ‘effective compute’. Many problems would be solved if we let an AI system have unlimited effective compute (for training and/or inference), but it isn’t clear such a system would excel at all kinds of intellectual tasks. For example, even if you had unlimited compute, it isn’t clear that current architectures and training environments would allow a system to succeed in domains which are lacking a clean algorithmic way of rating the quality of the outputs. Much of the recent progress in AI has relied on reinforcement learning with verifiable rewards (RLVR), but when there are no verifiable rewards, performance may plateau. If so, even a completed singularity in measures like efficiency and effective compute needn’t imply that the system is more capable than a human across the board after time t*.

In physics, scientists distinguish between a coordinate singularity and an essential singularity. The former exists when the singularity is an artefact of how things are being measured. For example, when Schwarzschild (1916) published his mathematical description of a black hole there was a mathematical singularity at the black hole’s event horizon. It took decades of scientific work before physicists were able to establish that this was a mere artefact of the coordinates being used (whereas the singularity at the centre of the black hole was not fixable by a change in coordinates).

We can adopt this distinction when studying RSI. For example, suppose we are measuring A by the mean time between failures (MTBF): how long the system can go before making a mistake while completing tasks of a certain kind. This is a standard measure in engineering and a relevant measure of capability for many real-world uses. In this case, one might see A rising towards a singularity. But consider an alternative measure B which is the percentage of times the system gets the right answer. If A goes to infinity at time t*, that just means B reaches 100% at t*. While a lot of systems never quite reach 100% reliability, there is nothing impossible or paradoxical about doing so. Thus in this example, t* is a coordinate singularity in measure A, but not in measure B, and is not an essential singularity.

Note that a very widely used measure of general AI capability — the METR time horizon (Kwa et al. 2025) — is somewhat like a mean time between failures. Recent years have seen exponential (or perhaps super-exponential) growth in time horizons for AI systems completing the kinds of routine professional computer-use tasks in the benchmark. But even if the time horizon went to infinity,11 it isn’t clear the system would be as intelligent as a human. It may still lack various cognitive skills (such as continual learning, sample efficiency, or creativity), which are important but not being tested. And there may still be substantial room for other AI systems to be much more capable, even at the skills that were being tested — systems that could solve much more challenging computer-use problems using much less compute. An infinite METR time horizon would just mean that the AI system can do the kinds of tasks in the benchmark with perfect reliability (and unbounded duration).

So even though most people studying intelligence explosions assume that the capability measure A won’t actually go to infinity in finite time, it wouldn’t be absurd or unphysical for it to do so. It depends on what measure it is. Some measures could go to infinity without incident. If such measures could also satisfy the key differential equation (or time-embedded difference equation) then one could get a completed intelligence explosion — A would have gone to infinity in finite time.

I’m not quite sure what we should think about this. It really depends on the nature of the measure, and this result is much easier to achieve on narrow measures — those that can rise unboundedly even when many key cognitive abilities are lacking. But measures like mean time between failures or METR time horizons seem unlikely to lead to rapidly decreasing generation times — what goes to infinity is a measure of duration, not a measure of speed. By showing that a completed singularity is theoretically possible, I’m mainly trying to caution people about the behaviour of certain intelligence measures, rather than trying to suggest we will genuinely have unbounded progress in intelligence in a finite time.

Let’s now turn to look at what happens when an intelligence explosion can’t get to infinity.

Going Finite

We’ve now seen how super-exponential growth can be cleanly divided into singular and sub-singular varieties, with the singular kind appearing very difficult to achieve (once we take the discrete nature of feedback loops into account). And we’ve seen that the rates at which f(A) has to grow or diminish are independent of which measure is used (so long as the relevant measures are ‘similar’ to each other).

But all of these claims rely on a clean mathematical model where growth is classified by its asymptotic behaviour as t or A approaches infinity. In the real world, this model may very well break at some finite value of t or A, giving increasingly inaccurate results thereafter. That’s a big problem for this kind of asymptotic analysis — as it relies on the infinite domain (or range) to produce its clean classifications. I hope that these divisions which are natural in the idealised setting will be natural in the real-world application. This is often the case,12 though it isn’t guaranteed.

There are a variety of finite limits we might expect to run into, which could force the clean mathematical model to break at different points, and in different ways.

The most widely discussed is a ceiling on intelligence: a horizontal line at some level A*, which A(t) cannot cross. In the study of feedback systems, people often say that A saturates as it approaches such a horizontal asymptote. It is generally thought that this will happen for RSI.

There are many different kinds of ceiling at which the AI’s intelligence (or efficiency) may saturate:

• Limits of intelligence itself — Even an optimal reasoner would be neither omniscient nor omnipotent. It would need to perform experiments to gain knowledge, could still be beaten at unbalanced games, and may face intractable prediction problems in chaotic or agentic domains.

• Limits of intelligence per unit resource — Even if we have the optimal algorithms and hardware, our solar system has only one star out of the 200,000,000,000 in our galaxy, and growth beyond our system is slow and cubic. If a galactic superintelligence would be more intelligent than a stellar superintelligence, it will be a long time before we could reach that higher level.

• Limits of the hardware paradigm — Even an optimal silicon chip may be far below the physical limits of compute per unit resource.

• Limits of the algorithmic paradigm — Even an optimal neural network may be far below the best intelligence that could be achieved with that amount of compute.

• Limits of training data — The training data we have (and could acquire during RSI) is lacking a lot of information on many domains (especially non-verbalisable information and contextual information). So we could have a very intelligent system that is limited by inability to train certain skills.

• Earlier limits — The ascent may stall out before any of the above due to limitations inherent in the starting AI.

Discussions of RSI sometimes tacitly assume that an intelligence explosion would only stop when it reached the limits of intelligence itself, and then use assumptions like omniscience or perfect rationality to model its behaviour. But an explosion might saturate significantly below this level, with important consequences for predicting post-explosion capabilities and for understanding whether all intelligence explosions need end at the same intelligence level.

The manner in which super-exponential growth slows down as it approaches its limit might be analogous to the way exponential growth runs out of steam. While the simple equation for exponential growth is Ȧ = kA, in reality there is usually an additional dampening term that starts small but grows to dominate for large A. For example if we subtract a quadratic term that has been shrunk down by a large factor (K), this gives the differential equation for logistic growth,

where A(t) approaches a horizontal asymptote at height K. We could use the same model for super-exponential growth:

Or more generally, we could assume Ȧ = f(A) − g(A), where g(A₀) ≈ 0 and g(A) < f(A) until some crossover point, A*. This will produce a horizontal asymptote at height A*.13

This dampening force could come from running out of room for improvement as a system approaches some form of perfection (such as optimal intelligence per unit resource), or it could come from something like straining under the growing size or complexity of the system. The latter could stop the ascent before reaching any kind of optimal system.14

As well as a ceiling on A, one could also run into upper limits on how quickly it can increase — either in absolute terms (such as a limit on Ȧ) or percentage terms (such as a limit on Ȧ/A​). Rather than making A(t) converge to a horizontal asymptote, a maximum gradient would make A(t) converge towards a diagonal line, while a maximum growth rate would make A(t) converge towards an exponential.

What could produce limits of these kinds?

One mechanism I find particularly likely is if there is a floor on the achievable generation time, T_n​. It seems very unlikely that generation times can be brought arbitrarily close to zero. There are many kinds of feedback loop that could contribute to RSI, ranging from decades (e.g. designing a successor to EUV lithography) to months (e.g. designing better pretraining) to seconds (e.g. designing better scaffolds). While some are extremely short, those only capture a tiny fraction of the pipeline of what makes for better AI R&D, so it seems likely they would quickly saturate if the other aspects of AI improvement were unchanged. For example, putting a fixed agent in a scaffold that has the agent repeatedly redesign that very scaffold might make some improvements, but without changing the model itself, it seems very unlikely to take off towards infinity. And while continued RSI might be able to reduce any of these generation times by a substantial factor, each one seems likely to run into constraints where a certain minimal amount of time has to pass in order to make an improvement.

What would a lower bound on the achievable generation time do? If T_n​ approached some minimal value (T*) while A_n​ grew by a fixed amount each loop (ΔA_n = k), then A(t) would approach a linear rate of increase. If A_n​ instead grew by a fixed proportion each loop (ΔA_n = kA_n​), then A(t) would approach an exponential rate of increase.

I think the second of these looks quite plausible. If so, we might see multiple phases of an explosion:

0. The initial exponential phase when the doubling time of A is driven by human-only research.

1. Increasing amounts of RSI drive the generation time down towards machine speeds, so the growth rate of A climbs from its human-only rate towards some very high fully automated rate. Because this phase involves a continuously increasing growth rate, it is super-exponential in shape.15

2. Fully automated RSI continues for a while at this faster exponential, but as A climbs, it starts to saturate. This phase would be exponential in shape.

3. As it approaches its inflection point and subsequent horizontal plateau, the trajectory has substantially departed from its exponential form and is revealed to be a logistic.

This trajectory involves two kinds of finite limits on the idealised process — first on generation time (and thus growth rate of A), then on the absolute level of A. It shows how we could have a super-exponential process (which would be worth modelling as such), yet it runs out of steam in two distinct ways. On this view, the mathematical models of super-exponential growth could be a good fit (and thus predictive) for the period where generation time is reducing towards zero in some clean way (e.g. 10% shorter each loop), but then cease to be a good model as we enter phase 2.

I’ve been assuming that A would be improving by a constant factor each time round the loop, guaranteeing super-exponential and logistic phases, but versions with other functions for f(A_n) are also possible and can still show distinct phases if generation time saturates prior to intelligence level saturating.

Conclusions

My aim in this paper has been to improve our theoretical understanding of intelligence explosions — looking at the most explosive possibilities (super-exponential and singular growth) and asking what conditions are required to produce them.

We first looked at models based on differential equations and saw that the key threshold for super-exponential growth of some measure A is when Ȧ grows superlinearly in A. But the threshold for singular growth is somewhat higher — Ȧ has to grow slightly faster than all functions of the form Alog⁡(A) log⁡(log⁡(A))... This allows for the possibility of ending up with growth that is super-exponential yet doesn’t approach a finite time singularity.

We then saw that when you take the duration of the feedback loop into account, singular growth becomes much harder — requiring this generation time to approach zero (and quickly enough). Indeed, this model helps us see that singular growth is more about reducing the generation time than it is about increasing how much each loop contributes. And the dynamic that produces singular growth is simply the combination of the Zeno condition and the boundedlessness condition. On this more realistic model, growth that is super-exponential without being singular looks much more likely. It is what you get whenever you can make ΔA grow super-linearly in A, yet can’t reduce the generation time all the way towards zero. So analysis of explosive growth needs to be careful to distinguish singular growth from merely super-exponential growth.

For example, suppose we saw increasing monthly growth rates in some key intelligence measure: 10% growth, then 20%, then 30%. From the economics-inspired literature, we might have thought that since this is super-exponential, it must be hyperbolic growth. But we now know increasing growth rates are not the signature of singular growth. (If the pattern above continued, A would ‘merely’ be growing as e^0.05t².)

The importance of generation time to the dynamics of intelligence explosions suggests that generation times need to be carefully measured and tracked. It may be a good policy idea to require frontier labs to report their current generation times — especially those for pretraining and for RLVR post-training.

Two of the most important implications for the measurement of an intelligence explosion are negative. When we move beyond the simple model of Ȧ = kA^r , you cannot tell whether a process will explode or not based on its returns over a finite period. This is because the condition for having a singularity is a global property of the relationship between Ȧ and A (the blow-up condition), rather than a local property such as elasticity > 1. The same is true when modelling it with time-embedded differential equations or delayed differential equations, where the Zeno and unboundedness conditions are inherently global. In all cases, deficiencies in growth somewhere can be made up for by more extreme growth elsewhere.

So without strong assumptions about the functional form, you can neither rule out nor confirm that an explosion is under way based on local measurements. So while we should probably take measurements of elasticity > 1 for f(A) over some range of A as evidence in favour (and elasticity < 1 as evidence against) that evidence is limited. We should also consider other forms of evidence such as how the elasticity has been changing, or how quickly — and how far — the generation time can be reduced.

While it is not unique to my modelling, I also want to stress that all the kinds of feedback loops we’ve explored have behaviours that are exquisitely sensitive to fine differences near the borderlines of their behaviours. In contrast, the empirical measurements of intelligence that we can perform are all quite rough and noisy. So if the measured values suggest the growth of f(A) or the shrinking of the generation time is anywhere near this border, it will be extremely hard to empirically determine the future behaviour. These limitations are important for companies setting their internal evaluations and for the prospects of regulation.

After examining what produces singular growth, we also looked at the choice of how we measure intelligence. We saw that whether intelligence is growing super-exponentially often depends on the nature of our measure, while whether it is growing towards a singularity is much less dependent on it. This fact means it isn’t crucial to decide whether to use some measure or the log of that measure — you would look for the same signature of how Ȧ is growing with A in both cases. But when one measure can rise to infinity without the other following suit, the choice of measure really matters. In particular, some popular measures like METR time horizons are probably not the right tool for the job, since they can go to infinity without bringing other measures of intelligence along with them.

And we looked at how various finite limits might bite first, preventing truly singular growth. As well as the familiar idea of intelligence saturating at some maximal level (or maximal given various constraints) a minimal generation time could also derail the models of singular growth far before we reach that point.

Finally, while I’ve argued that singular growth is harder than we may have thought, that doesn’t mean RSI is safe or that AI R&D will move at a manageable pace. RSI might be able to speed AI R&D up to dangerously fast speeds even just with a linear speed-up. For example, if the human-only trajectory were A(t) and RSI sped this up to A(10t), we’d be getting a decade of human-only progress each year, introducing many of the dangers — even without any change in the fundamental shape of the curve.

With thanks to Tom Davidson, Rohin Shah, Loren Fryxell, Fin Moorhouse, Matthew van der Merwe, Will MacAskill, Robert Trager, Simon Biggs, and Shamil Chandaria for helpful discussions.

Appendix: Table of Rates of Growth

References

Philippe Aghion, Benjamin F. Jones, and Charles I. Jones. (2017). ‘Artificial Intelligence and Economic Growth’, NBER Working Paper 23928.

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas. (2016). ‘Learning to learn by gradient descent by gradient descent’, https://arxiv.org/abs/1606.04474

Anthropic. (2026). System Card: Claude Mythos Preview [System Card]. https://www.anthropic.com/claude-mythos-preview-system-card

Nicholas Bloom, Charles I. Jones, John Van Reenen, and Michael Webb. (2020). ‘Are Ideas Getting Harder to Find?’, American Economic Review, 110(4):1104--44.

Nick Bostrom. (2014). Superintelligence: Paths, Dangers, Strategies, Oxford University Press.

David J. Chalmers. (2010). ‘The Singularity: A Philosophical Analysis’, Journal of Consciousness Studies 17:7-65.

Alan Chan, Ranay Padarath, Joe Kwon, Hilary Greaves, Markus Anderljung. (2026). ‘Measuring AI R&D Automation’, https://arxiv.org/abs/2603.03992

François Chollet. (2019). ‘On the Measure of Intelligence’. https://arxiv.org/abs/1911.01547

Tom Davidson. (2025a). ‘How Can AI Labs Incorporate Risks from AI Accelerating AI Progress Into Their Responsible Scaling Policies?’.\ https://www.forethought.org/research/how-can-ai-labs-incorporate-risks-from-ai-accelerating-ai-progress-into

Tom Davidson. (2025b). ‘Will the need to retrain AI models from scratch block a software intelligence explosion?’.\ https://www.forethought.org/research/will-the-need-to-retrain-ai-models

Tom Davidson, Jean-Stanislas Denain, Pablo Villalobos, and Guillem Bas. (2023). ‘AI capabilities can be significantly improved without expensive retraining.’\ https://arxiv.org/abs/2312.07413

Tom Davidson, Rose Hadshar, and Will MacAskill. (2025). ‘Three Types of Intelligence Explosion’ https://www.forethought.org/research/three-types-of-intelligence-explosion

Tom Davidson, Basil Halperin, Thomas Houlden, and Anton Korinek. (2026). ‘When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks’, NBER Working Paper No. 35155.

Tom Davidson and Tom Houlden. (2025). ‘How quick and big would a software intelligence explosion be?’. https://www.forethought.org/research/how-quick-and-big-would-a-software-intelligence-explosion-be

Ege Erdil, Tamay Besiroglu, and Anson Ho. (2024). ‘Estimating Idea Production: A Methodological Survey’, SSRN. http://dx.doi.org/10.2139/ssrn.4814445

Daniel Eth and Tom Davidson. (2025). ‘Will AI R&D Automation Cause a Software Intelligence Explosion?’. https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion

A. Fawzi, M. Balog, A. Huang et al. (2022). ‘Discovering faster matrix multiplication algorithms with reinforcement learning’. Nature 610:47--53. https://doi.org/10.1038/s41586-022-05172-4

I. J. Good. (1965). ‘Speculations Concerning the First Ultraintelligent Machine,’ In F. Alt & M. Rubinoff, Advances in Computers (volume 6). Academic Press.

Demis Hassabis, Dario Amodei, and Zanny Minton Beddoes. (2026). The Day After AGI. World Economic Forum [Panel]. https://www.weforum.org/meetings/world-economic-forum-annual-meeting-2026/sessions/the-day-after-agi/

Charles I. Jones. (1995). ‘R&D-Based Models of Economic Growth,’ Journal of Political Economy 103(4):759--84.

Andrej Karpathy. (2026). ‘Autoresearch’. https://github.com/karpathy/autoresearch

Michael Kremer. (1993). ‘Population Growth and Technological Change: One Million B.C. to 1990’, The Quarterly Journal of Economics 108(3):681--716.

Daniel Kokotajlo and Eli Lifland. (2025). ‘Takeoff Forecast’ https://ai-2027.com/research/takeoff-forecast

Raymond Kurzweil. (2001). ‘The law of accelerating returns’. https://www.writingsbyraykurzweil.com/the-law-of-accelerating-returns

Ray Kurzweil, Vernor Vinge, and Hans Moravec. (2003). ‘Singularity math trialogue’. https://www.thekurzweillibrary.com/singularity-math-trialogue

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan. (2025). ‘Measuring AI Ability to Complete Long Software Tasks’. https://arxiv.org/abs/2503.14499

Eli Lifland, Brendan Halstead, Alex Kastner, and Daniel Kokotajlo. (2026) AI Futures Model. https://www.aifuturesmodel.com

Hans Moravec. (1999). ‘Simple equations for Vinge’s technological singularity’. https://frc.ri.cmu.edu/\~hpm/project.archive/robot.papers/1999/singularity.html

Hans Moravec. (2003). ‘Simpler equations for Vinge’s technological singularity’. https://frc.ri.cmu.edu/~hpm/project.archive/robot.papers/2003/singularity2.html

Anders Sandberg. (2013). ‘An Overview of Models of Technological Singularity’. In The Transhumanist Reader (eds M. More and N. Vita-More). https://doi.org/10.1002/9781118555927.ch36

Karl Schwarzschild. (1916). ‘Über das Gravitationsfeld eines Massenpunktes nach der Einsteinschen Theorie’. Sitzungsberichte der Königlich Preussischen Akademie der Wissenschaften. 7: 189--196. Translated as, Antoci, S.; Loinger, A. (1999). ‘On the gravitational field of a mass point according to Einstein’s theory’. https://arxiv.org/abs/physics/9905030

Ray J. Solomonoff. (1985). ‘The time scale of artificial intelligence: reflections on social effects’. North-Holland Human Systems Management 5:149--153.

Philip Trammell and Anton Korinek. (2023). ‘Economic Growth under Transformative AI,’ NBER Working Paper 31815, https://doi.org/10.3386/w31815.

Lizka Vaintrob and Owen Cotton-Barratt. (2025). ‘AI Tools for Existential Security’. https://www.forethought.org/research/ai-tools-for-existential-security

Eliezer Yudkowsky. (2001). ‘Creating friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures’, The Singularity Institute.

Of past algorithmic improvements which decrease LLM pretraining loss, many have been “scale-dependent” – that is, improvements which decrease LLM pretraining loss by more, relative to prior algorithms, at larger quantities of compute.

The influence of such scale-dependent algorithms is (1) moderate evidence that attempts to limit algorithmic progress in absence of compute limitations will be ineffective, (2) weak evidence that our inferences about the future scale of algorithmic progress, based on the past, are invalid, and (3) part of a plausible argument either for or against a software intelligence explosion, depending on other details of one’s model.

I’ll proceed by discussing the following:

1. How scale-dependent algorithms for pretraining loss probably account for the majority of algorithmic improvements in this domain

2. How scale-dependent algorithms for end-to-end task performance might account for the majority of algorithmic improvements in this domain

3. How scale-dependent algorithms in the past should influence our model of algorithmic improvements in the future

4. Possible policy implications

Before starting, a note on method – when discussing questions regarding AI’s past and future progress, there’s a natural continuum from “the intractable, high-impact, abstract question, which we terminally care about” to “the tractable, dubious-impact, concrete question, which we instrumentally care about.”

Consider questions about scale-dependent algorithmic improvements, sliding from the former to the latter along such a continuum.

• We are most terminally interested in questions about the future: “Will future improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? What can we know about how these two will be causally intertwined? Will future improvements enable a software-only intelligence explosion – that is, in a world with total compute held constant, could algorithmic progress lead to an intelligence takeoff in a comparatively short period of time?”

• The data we can gather most relevant to this question is about the past: “Did past improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? Would past AI algorithmic improvement have counterfactually looked slower, without the datacenter buildout?”

• And finally we have the narrower sub-question for which we have the most past data, which contributes to answering the broader historical question: “Did past improvements to LLM pretraining loss depend more on compute scale-up or on algorithmic improvements?”

I’m going to start by discussing the evidence for this last question, before moving to the increasingly impactful and increasingly difficult earlier questions.

Scale-Dependence of Algorithmic Improvements for Pretraining Loss

Are the algorithmic improvements that have most decreased pretraining loss for LLMs scale-dependent or scale-independent? What would either of these mean?

The compute-equivalent-gain (CEG) for some algorithm A’, relative to another algorithm A, is the ratio of the FLOPs needed to train an A-using model to a given level of performance, over the FLOPs needed for an A’-using model to reach that same level. If it takes 3x as much compute to train an LLM with algorithm A to reach the same pretraining loss as another LLM trained with algorithm A’, then A’ has a 3x compute-equivalent gain on pretraining loss relative to A.

Given the above notion of CEG, algorithmic improvements that produce some kind of CEG could conceivably be scale independent or scale dependent.

A scale-independent algorithm has a constant CEG across different quantities of training FLOPs. So if A’ gives a 2x multiplier relative to A at about 10¹⁵ FLOPs, it continues to give a 2x multiplier at about 10²⁵ FLOPs. Correspondingly, if one were somehow able to know antecedently that some algorithm was scale independent, one would just need to test it at one scale to find whatever this constant happened to be.

By contrast, a scale-dependent algorithm has a variable CEG, which is a function of the number of FLOPs used in training. So A’ might give a 2x multiplier relative to A at about 10¹⁵ FLOPs, but give a radically different 1000x multiplier at about 10²⁵ FLOPs. If one were somehow able to know antecedently that some algorithm was scale dependent, one would know that one needed to test it at many scales to find the function mapping from FLOPs to the compute multiplier.

The thesis of 2025’s “On the Origin of Algorithmic Progress in AI” is that the vast majority of apparent algorithmic progress in LLM pretraining loss per-FLOP has come from a handful of scale-dependent algorithms. So, according to the paper, the apparently steady flow of “algorithmic improvements” in the past came from choosing a reference point algorithm (A) prior to some particular scale-dependent algorithmic improvement (A’), and then mistaking the continued improvements A’ makes over A as compute increases for the presence of additional algorithms, B, C, and so on. In the absence of an increase of FLOPs, this apparent radical increase in performance would not have been nearly as large.

What kind of evidence is there that the majority of progress in LLM pretraining loss per-FLOP actually does depend on scale-dependent algorithms?

• Well, the paper presents good evidence that scale-dependent algorithms are very important.

  • LSTM to Transformer Change: “On the Origin” uses experiments to estimate how much the Transformer improves over the LSTM as compute scales up. At ~10¹⁵ FLOPs, this comes to about 6x CEG; at ~10¹⁷, it comes to a 28x CEG increase; at the level of compute with which the Transformer was introduced in 2017, they extrapolate it to about a 91x CEG increase.

  • Kaplan to Chinchilla Change: “On the Origin” grants that Chinchilla scaling is correct, then projects forward the gains you get by switching to Chinchilla from Kaplan; they are approximately equivalent at 10²⁰, and give a ~10x or more gain at frontier scales.

• The paper also presents evidence that scale-independent algorithms are less important than one might imagine.

  • Their ablations find that scale-independent additions are not multiplicative. That is, doing a bucket of changes that move one from the “retro” Transformer to the modern Transformer in theory should give a CEG of 3.5x, if one were to multiply the gains that the additions make separately; but in practice only gives 1.3x.

  • The modern transformer similarly does not improve greatly on the “retro” Transformer as you scale up.

So “On the Origin” projects that, given the actual historical increase in FLOPs that makes scale-dependent algorithms more efficient, one observes a 21,400x CEG increase from 2017 to 2025. But if one counterfactually holds compute constant at 10^18 FLOPs, this amounts to only a 155x CEG increase. This amounts to the difference between a ~7 month doubling time for algorithmic efficiency (very close to Ho et al. (2024)’s 8-month doubling time); and a ~13 month doubling time of algorithmic efficiency. But note that the second “doubling time” here is specifically relative to the level of compute at which one calculates the increase in algorithmic efficiency; there would be a different doubling time for a different absolute quantity of compute.

So there is a (very approximate) halving of the speed of algorithmic improvement, given that you hold FLOPs fixed at 10¹⁸, once you take into account the difference between scale-dependent and scale-independent algorithms. And of course, this ~13 month doubling time is still fueled almost entirely by the gigantic LSTM to Transformer transition at 10¹⁸ FLOPs – leaving this quantity of algorithmic progress out means that the doubling time gets vastly larger.

This is a pretty large change to the proposed rate at which constant-FLOP algorithmic improvements occur. But you can actually make a case that “On the Origin” understates the degree to which algorithmic improvements in LLM pretraining loss depend on scale-dependent algorithms.

The largest single purportedly scale-independent gain (2x) described in the paper belongs to mixture-of-experts (MoEs), an alteration to the Transformer architecture. But several subsequent works have shown that the gain from MoEs over dense architectures is actually strongly scale-dependent, and probably larger than 2x. “Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” estimates a 7x gain at 10²² FLOPs. “Scaling Laws for Fine-Grained Mixture-of-Experts” estimates a 20x gain at 10²⁰ FLOPs. In both cases, the size of the CEG relative to a dense model keeps improving with the quantity of FLOPs. Although it’s uncertain which exponent is correct, both papers point toward MoEs being a scale-dependent algorithmic improvement that substantially improves with the quantity of FLOPs.

So – to return to the motivating question – most of the algorithmic improvements to LLM pretraining loss, so far, have probably depended on a compute scale-up. Apparent algorithmic progress in pretraining loss would have been much smaller if total compute had been held constant.

Scale-Dependence of Algorithmic Improvements for End-to-End Performance

No one actually cares about pretraining loss for its own sake, though.

“Decreased pretraining loss” is just one way to improve end-to-end performance. There are also post-training innovations like RLHF or best-of-N that improve task performance. There are also changes to data mixtures that improve performance. And of course RLVR-over-CoT has probably been by itself responsible for the greatest single leap in AI performance of the last few years.

How much are algorithmic improvements to end-to-end performance scale-dependent? In particular, how much does RLVR-over-CoT depend on scale-dependent algorithmic improvements?

This is deeply uncertain. There’s certainly no work that provides a clean FLOPs-to-CEG function, as in the case of pretraining loss.

There are several reasons for thinking that RLVR works extremely poorly or not at all beneath a certain scale, and so is somewhat scale-dependent. You need a certain success rate for reinforcing successes to be viable. Several works remark on how RLVR without sufficient scale simply doesn’t teach the required diversity of behaviors, and that RLVR on small models plateaus far lower than RLVR on larger and longer-trained models. So there’s suggestive evidence that at the frontier RLVR is scale-dependent, but nothing conclusive.

It would also be difficult to perform an experiment to cleanly determine whether RLVR-over-CoT is scale-dependent for several reasons. For instance, the importance of RL, in general, probably comes from how it permits at least somewhat out-of-distribution generalization, rather than from how it improves performance on in-distribution tasks. But to my knowledge, there’s no publicly proposed clear way to measure out-of-distribution generalization; so there’s no measure against which one could test how RLVR improves with scale on the matter where it ostensibly has the greatest impact. Similarly, RLVR takes place as the final stage of training, after a large sequence of data-selection, pretraining, and midtraining, which means there are just many more free variables it would be necessary to consider before deciding on an experimental design.

Apart from RLVR, there’s some evidence that some apparent scale-independent improvements may scale negatively with FLOPs, and thus provide few or limited gains at the frontier. For instance, the paper “A Bitter Lesson for Data Filtering” claims that although filtering data improves performance at a small amount of compute, it scales negatively with FLOPs, such that with enough computing power it’s best to do no data filtering at all.

I think it’s rather likely that RLVR and other important post-training techniques improve in a strongly scale-dependent way, and some of my analysis below will lean on that belief. I’d be surprised if RLVR doesn’t improve in a somewhat scale-dependent way.

But I don’t think there’s any conclusive empirical evidence here, and to the degree I’m wrong, some of my inferences below are less likely to be true.

How scale-dependent improvements in the past impact our model of “algorithmic progress” for the future

Let’s grant that, in the past, scale-dependent algorithms have been responsible for the majority of algorithmic improvements in both LLM pretraining loss and in end-to-end task performance.

What consequences would this have for how we expect algorithmic progress to go in the future? What consequences does this imply about a future where compute is “held constant,” as in the case of a software intelligence explosion? Would it lead us to expect a radical trend break, in either direction?

Let’s take the other side by contrast, first – imagine a model of the world where almost all algorithmic improvements are scale independent: improvements are drawn from a pool of possible improvements, and the number and efficacy of the algorithms taken from the pool do not change merely by the fact of compute scaling up.

If you operate off this model, then there are some natural consequences.

• Algorithmic improvements do not become “more discoverable” as FLOPs scale up. More compute may make it easier to run experiments to find improvements, but scaling up the FLOP count within each experiment does not make finding the right algorithm easier or harder per-experiment.

  • There could, of course, be other factors that make algorithmic improvements easier or harder to find. One improvement could open up a line of research that leads to other improvements, making it easier; it might decrease the total pool of improvements, making it harder. But the fact of having scaled up FLOPs alone does not make improvements easier or harder to find.

• Some particular algorithmic improvement does not become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it would equally well give a 3x CEG if it were found two years into the future.

• Broadly, our actual historical rates of “algorithmic progress” reflect something like the rate at which algorithms are naturally found, in a way indifferent to compute build-out.

By contrast, one can model a world where most algorithmic improvements are scale dependent as a world where improvements are drawn from a changing pool of FLOP-specific algorithmic improvements. Pools get “unlocked” as the compute frontier moves past the point where they start to become significant, and the pools grow progressively more explored over time as more and more researchers have access to the compute necessary to explore them. So both “how explored” each pool is, and the consequences of finding an entry from each pool, are always changing.

If you operate off this model, then:

• Algorithmic improvements become much “more discoverable” as FLOPs scale up; the more FLOPs you use in an experiment, the higher the relative CEG might become, and so the easier the new algorithm will find it to overcome the “well-tuned baseline” algorithm with very well-adjusted hyperparameters. So in addition to letting you run more experiments, additional compute may enhance “research taste.”

• Some particular algorithmic improvement might actually become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it might give a 100x CEG in the future when applied with more FLOPs, if its discovery is delayed.

• Broadly, this view implies that “algorithmic progress” depends on past compute scale-ups, and would be much slower without them.

Imagine that one is trying to determine how much algorithmic progress is likely in the future, granted that one is in a scenario where total compute is held constant. It’s important to note that switching to a view of algorithmic progress as scale-dependent might move your estimate of the likely rate of algorithmic progress in a constant-compute regime either up or down.

The case for moving it down is clear. Switching to a view where scale-dependent improvements are significant could move one’s estimate of algorithmic progress down, because it moves one estimate of past algorithmic progress in a constant-compute regime down. One used to estimate a ~8 month halving time to reach equivalent loss for equal FLOPs; now one estimates a ~13 month halving time; and the future is likely to be like the past, so future FLOP-constant algorithmic progress is likely to be slower.

But switching to such a view could move one’s estimate of algorithmic progress up, because it decreases your credence that the future will be like the past in this respect! If different algorithms get unlocked at different levels of compute, then potentially some future yet-to-be-unlocked regime of algorithmic improvements might differ enormously from past algorithmic improvements. There’s likely less reason to think that the number or importance of algorithmic improvements available in one prior algorithmic bucket will be equally important as algorithmic improvements in subsequent buckets as compute increases.

So if one finds such a picture compelling, then the likelihood one assigns to a SIE could go up. If there is a future compute scale-up to 10³² FLOPs, a SIE taking place then might find many algorithms that were totally unavailable to us in the 10²⁷ regime.

Overall, all this remains uncertain. I think various philosophical arguments suggest that there’s a greater diversity of room for algorithmic progress as compute scales up. The human brain is largely a linearly scaled-up primate brain, but individual humans and humans in aggregate can do many calculations no non-human primate can. And there are likely greater gains from organizing 10,000 humans well than from organizing 10 humans well.

Possible Policy Implications

If a majority of algorithmic improvements to task performance consistently come from scale-dependent algorithms, and are likely to keep doing so in the future, how would this impact policy interventions into AI?

There are two notable consequences, which mirror each other: (1) efforts to decrease algorithmic progress, in absence of a training compute scale-up, would be less necessary than previously believed; and (2) efforts to decrease algorithmic progress, in the presence of a training compute scale-up, would be less effective than previously believed. I will discuss them each in turn.

First, some policy interventions meant to slow or stop the development of advanced AIs have focused on limiting the size of the largest training runs. In this context, a steady advance of scale-independent algorithmic improvements would have decreased the efficacy of any particular maximum training-run size limitation. For instance “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence” recommends a maximum training-run size of 10²⁴, but also notes that the “number of operations used to train an AI to a given capability level drops by 3x each year.” Given this trend, it’s easier to derive the need to prohibit AI algorithmic research, a measure that the paper itself notes may be “controversial and normally a bad idea,” but which it believes to be necessary to avoid the creation of ASI.

But if the majority of algorithmic improvements have been scale-dependent, then prior estimates for scale-independent algorithmic improvements are too high; and so prior work gives a less-good reason for thinking that the number of operations used to train an AI to a given capability level drops by 3x every year. So the need to ban and monitor research would be correspondingly decreased; a hard cap on maximum training-run size would be more effective than previously estimated.

Second, some proposed policy interventions have focused on scaling up the size of compute while trying to decrease or hold constant algorithmic progress. AI 2040, for instance, proposes continuing to increase the size of training runs from 10²⁶ now to 10³² in 2035, while decreasing the rate of algorithmic progress.

Overall, if you believe that the majority of important future algorithmic improvements will be scale dependent in the way that the majority of past algorithmic improvements were scale dependent, that should decrease the credence you give to our ability to “not find” these improvements. Why?

As mentioned above – as compute scales up, the increasing gains from scale-dependent improvements will make them more obvious. By analogy to a counterfactual world in which the Transformer was never discovered, an incredibly poorly-done, badly-implemented Transformer in 2026 still obviously improves on an LSTM, because even an awful implementation might mean it is only 100x better rather than 1000x better than the LSTM. It’s probably impossible to present the Transformer from being discovered for too long. Similarly, continuing to scale up compute makes scale-dependent algorithms more obvious; it probably decreases the quantity of research taste you need to discern them.

The other part of the reason is just because, well, continuing to scale up compute keeps opening up new possible unseen algorithmic buckets that allow improvement; the space of algorithmic improvements will keep growing, different kinds of research taste will find more progress.

This article was created by Forethought. See all our research on our website.

Peter Salib is an Assistant Professor of Law at the University of Houston Law Center and co-director of the Center for Law & AI Risk. Simon Goldstein is an associate professor at the University of Hong Kong, a senior editor at AI Frontiers, and a visiting senior scholar at Forethought. They have written together about AI safety, AI rights, and the governance of advanced AI.

They joined Forethought’s Fin Moorhouse to discuss their new paper, Liberalism Forever. The conversation covers:

• Their “null hypothesis”: today’s liberal institutions — free markets and democratic governance — may already hold most of the answers for governing the far future, and longtermists have underrated their power

• Examining whether canonical social-science arguments for markets and democracy still hold under transformative AI, space colonization, and explosive growth, and claiming that most survive (and several get stronger)

• Why they favour reasoning “at the margin” over designing a detailed end-state, and why proposals like viatopia and the long reflection sound liberal but risk illiberalism when actually implemented

• Whether the standard case for markets — information aggregation, allocative efficiency, innovation — still bites once AGI can read off preferences directly

• Why inequality is, in their view, a comparatively boring problem with a known solution (tax-and-transfer), why AI-specific taxes like a compute tax are misguided, and how the growth-versus-distribution trade-off changes when growth rates get very high

• Dyson swarms and space resources: why they doubt there’s any real monopoly worry, since energy would be additive with free entry, and why property auctions might beat egalitarian allocation schemes

• The arguments for democracy that persist post-AGI — public-choice/selectorate incentives and democracy as a commitment device — versus the epistemic arguments, which they think weaken

• Why AGI could make autocracy more competitive (loyal AI bureaucrats and militaries removing the need for a human winning coalition), and what preserving democratic control requires in response

• Handoffs to superintelligent AI and successionism, and their preferred alternative of extending the franchise to AIs with their own values rather than deferring wholesale

• The history of new “technologies of democracy,” from the Federalist Papers to LLMs as impartial arbiters, and where each author thinks their own argument is weakest

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Will MacAskill is a senior research fellow at Forethought and the author of What We Owe The Future. Tom Davidson is a senior research fellow at Forethought and the author of a series of reports on AI timelines, takeoff speeds, and AI-enabled coups.

In this informal conversation about early-stage research in progress, they discuss:

• Why a philanthropist who takes transformative AI seriously should expect extraordinary investment returns (plausibly 10x to 100x) and how the market doesn’t seem to be pricing this in

• Why AGI shouldn’t be treated as a hard deadline to spend by, and the case that most philanthropic money should be spent during the intelligence explosion rather than before it

• How genuine AI substitutes for human labor would dissolve the hiring bottleneck that makes organizations so hard to scale today

• How relative prices shift when cognitive labour becomes abundant, why the patent system stops making sense in that world, and the window of “crazy philanthropic bargains” open to actors who adapt faster than large institutions

• The prospect of a world where only a small elite has access to superintelligent medical, financial, and strategic advice, and what it would take to avoid this

• Counter-considerations to the main case, e.g., crowding out by new money, the risk that overspending attracts grifters or distorts the field, and whether a richer world will mean worse philanthropic opportunities

• The risk of value drift over a long saving period, and mechanisms for binding your future self (such as irrevocably handing off control of philanthropic funds to independent, constitutionally-bound foundations)

• “Differential intellectual development”: using directable AI researchers to accelerate biodefense, infosecurity, AI interpretability, and neglected armchair fields like population ethics and social choice theory

• Why the case for waiting to give is much weaker for small donors

• Will’s worry that the whole line of reasoning might fail if it turns out scaling philanthropy well is bottlenecked by serial real-world learning that no amount of cognitive labor can compress

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Summary

• Governments are going to deploy frontier AI in the domains where they have exclusive powers: military, intelligence, surveillance, and punitive law enforcement.

• The existing checks on those powers are weak. AI deployment could weaken them further.

• Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention), legislatures often lack access and capacity, and, as AI replaces humans in government, nobody will be left to refuse or slow-roll unlawful or anti-democratic orders.

• In the limit, this can enable full-blown coups; in the short term, it undermines checks & balances.

This post lays out the work I’m most excited about:

1. Governance of AI applications in government: policies on how AI should be procured and used in government, research on how to govern AI agents in particular, and field-building beyond the EA/safety crowd.

2. Interbranch oversight: transparency and oversight provisions, audit tech for an automated government, waking up lawmakers, and increasing AI uptake in the legislative and judicial branches.

3. Developing AI for government use: model specs/constitutions that say something meaningful about behavior in government contexts, and evals for the qualities we’d want such systems to have (such as epistemic integrity, non-sycophancy, lawfulness).

4. Strengthening civil society: monitoring and freedom of information infrastructure for government AI use, a coalition of ML researchers at frontier labs, and tools that preserve citizens’ ability to coordinate (AI-for-epistemics + privacy tech).

Many ideas are more tractable than they might seem: specs are being written now, must-pass bills come around every year, and a lot of the civil-society work just requires someone to start doing it.

If you’re interested in working on any of it, apply to EIP’s recent RFP. You could also fill out this expression of interest form.

Why care about responsible government deployment of AI

How could government deployment go wrong?

• Governments will increasingly use the most powerful AI systems in domains where they have exclusive powers; particularly military, law enforcement, intelligence, and surveillance. This is most salient for the US government: Claude was reportedly already involved in the Maduro raid in Venezuela and assisted with targeting during the military engagement with Iran. With the recent Executive Order, at least some parts of the US government will have privileged access to frontier AI models for up to 30 days before release to other trusted partners.

• Without appropriate oversight or constraints, this could ultimately lead to a literal AI-enabled coup where a commander uses military AI systems to seize control of a country. Less extreme scenarios still undermine important checks and balances.

• This risk strikes me as similarly important as the risk from misalignment and there are way fewer people working on it. And importantly, alignment is not sufficient: even a perfectly aligned AI loyally serving a coup-staging principal is catastrophic.

Why is this not going to go well by default?

• Oversight into these government domains is already very limited. Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention) and legislatures often lack access and capacity. Activities in these domains are often classified/secret.

• As AI systems replace human bureaucrats, military officers, and soldiers, this will further empower the government and remove some of the remaining checks:

  • Governmental accountability partially rests on citizen bureaucrats and soldiers who refuse or slow-roll orders that are clearly unlawful or undemocratic. In 2024, for example, various commanders in the South Korean military refused orders during a constitutional crisis, which probably prevented a greater catastrophe. Unconstrained or personally loyal AI systems could change that.

  • The power and reach of government have historically been constrained by the friction inherent in a large bureaucracy. AI systems could remove that friction and make it much easier for leaders to impose their will on a country, which could make it much easier to abuse their position.1

  • AI could enable pervasive surveillance & cheap data processing, which could allow prosecution of political enemies on an unprecedented scale.

What to do about it?

• In the spirit of inspiring more action, I’ve compiled the efforts I’m most excited about below.

• If you’re interested in working on this topic, I’d encourage you to apply to EIP’s recent RFP or to fill out this expression of interest form. You should probably also take a look at this research agenda.

Rules for AI in government

Legislatures should set rules for the procurement and deployment of AI in government, especially for the most sensitive applications of government power (military, intelligence/surveillance, and punitive law enforcement).2

I think this is actually tractable right now, at least in the US:

• The recent Anthropic/DOW conflict has put this on the Congressional map for autonomy in weapon systems & surveillance.

• While Congress is passing fewer and fewer bills each year, there are annual bills for the intelligence community and military that have to be passed (IAA, NDAA), and there’s a lower bar for including stuff in them.

• In the future, crises or scandals will create legislative openings, and we should have proposals ready to go.

Work I am most excited about

Policy & advocacy: Develop and push for rules for AI systems in critical domains (or adjust existing ones to account for the use of AI systems). The most salient ones in my mind are:

• Autonomous weapon systems: Currently, they are only governed by a DOD Directive, which could be rescinded at any point. I’d love for more people to think through rules that would make it harder to use these systems against domestic opposition (e.g., technical guardrails against abuse, limiting the amount of autonomous force any one human can command, multi-party authorization schemes, geofencing).

• General-purpose systems in the military: They are currently not specifically regulated at all, but ultimately, they could orchestrate large-scale cyber attacks or command entire battalions. At that point, it will matter a ton whether they have any constraints at all.

• Surveillance: AI will enable data processing on an unprecedented scale. It’s worth thinking through how to adjust current rules in light of that (e.g., closing the data broker loophole).

Policy & advocacy: Develop and push for cross-cutting requirements for AI systems in government (general-purpose ones in particular).

• There is already an Executive Order requiring all AI systems in government to be truth-seeking and ideologically neutral. Those seem like great requirements to me that could be refined/improved and put into statute.

• Another cross-cutting proposal is law-following AI, which roughly says that AI systems in government should be trained to follow the law rather than just instructions. I’m excited for more people working out the open questions of this agenda, e.g., which laws should they be required to follow, how should they decide whether a contemplated action is likely to violate the law, in what contexts should the law require that AI agents be law-following?

• More minimally, AI systems could be required not to assist in undermining the constitutional order (though how to assess this is challenging in its own right).

Field-building: Build a broad-tent coalition for responsible deployment of AI in government.

• Responsible deployment and oversight is something that lots of people and political parties should be in favor of. It could be good to build a coalition that is nonpartisan and includes AI safety advocates, constitutional lawyers, civil liberties orgs, defense intellectuals, and national security think tanks.

Oversight of AI in government

Rules need to be enforced and updated, even as the technology changes. That will require the legislative and judicial branches to be institutionally and technologically empowered.

Again, there are some reasons for optimism in the US:

• Congress may soon be controlled by the opposite party from the Presidency, which creates incentives for increased oversight.

• Crises or scandals can open up opportunities for legislation (e.g., FISA following the Church Committee).

• Some non-legislative projects could still create significant value.

Work I’m most excited about

Policy & advocacy: Develop and push for the transparency & oversight provisions.

• We will need democratic oversight of AI systems, especially around surveillance and the application of force (in military and law enforcement). This is currently not in place and not on track to happen.

• Here are proposals I’m currently most excited about (in the US):

  • Requiring comprehensive logging of government use of AI (and designating such logs as federal records)

  • Automated flagging of suspicious AI behavior to oversight bodies (ideally, something like FISA courts or Congressional committees)

  • Whistleblower reform for national security domains (especially allowing whistleblowers to talk to Congress)

  • Increasing GAO’s capacity to audit deployment of AI in national security domains

Auditing tech: Pilot and mature the infrastructure required for overseeing widespread AI use.

• As AI is increasingly used in government, it will be both a challenge and an opportunity for oversight. The speed of AI will make it harder for regular human oversight to keep up, but AIs are in principle much more auditable than humans because they think out loud and automatically create records that can be scrutinized by other AI systems that flag suspicious behavior.

• There are already attempts to audit companies deploying large amounts of AI labor. I would love to see these piloted, matured, and adapted for classified government contexts.

Advocacy: Wake up lawmakers to the powers and challenges of AI.

• Some of the proposals in this doc require ambitious changes to the way government is run. They are more likely to happen if lawmakers are aware of the stakes involved.

• This could involve building trusted relationships, showcasing AI capabilities through easy-to-understand demos, and making insights from the AI safety community intelligible to people with much less context.

Products/programs: Increase AI uptake & unblock automation in legislative and judicial branches.

• To provide meaningful oversight, it will become increasingly important to understand and utilize AI systems. I’d love for lawmakers and judges to be well-versed in these systems.

• Examples of the things that I have in mind here include: providing training, building custom tools, bringing in technical experts through fellowships (e.g., TechCongress), and reconstituting the Office of Technology Assessment in Congress.

Policy: Coup-proof government-led AI projects.

• If there’s ever a government-controlled AI project (e.g., public-private partnership), its shape will matter a lot. It will probably be designed in response to a crisis, which favors existing templates & plans. I’d like for somebody to create one that carefully distributes power (e.g., multi-stakeholder governance boards, oversight mechanisms that work under classification constraints, sunset clauses, mandatory external audits, access guarantees so one company doesn’t end up controlling the stack).

Research (speculative): Write new constitutions.

• There is a chance the disruption caused by AI will create “constitutional moments”. At that point, many government functions may need to run at machine speeds: voter input, legislative deliberation, judicial review, law enforcement, or application of lethal force. How do we do that in a way that still preserves important democratic values, if that’s desirable? I don’t really have great answers to these questions, and would love for people with the right macrostrategy skill set to think about them.

• (Of course, the more of these problems we can address without actually requiring constitutional changes, the better. Constitutional changes are hard!)

Developing AI for government

There are many free parameters when it comes to the question of how to build AI systems in government and what constitutes safe and responsible AI systems in critical domains (setting aside whether companies can constrain government use through contracts).

Again, I think progress on this front is pretty tractable:

• Specs/constitutions for frontier models are being developed right now.

• Systems are already being procured by governments across departments, and that will probably only increase.

Work I’m most excited about

Research: Design and stress-test model specs/constitutions for government use.

• General-purpose models deployed in government should have some guardrails, e.g., they should not assist in staging coups to overthrow the civilian government. This is a tough line-drawing exercise where you don’t want models to overrefuse, but you also want some meaningful constraints. That makes me keen for people to work out desired behavior for lots of edge cases and create broad public buy-in for a set of minimal constraints.

Research: Build evals & datasets for desirable qualities in government-deployed AI systems.

• The US government has already said it wants AI systems to be truth-seeking and ideologically neutral. That seems great to me. Other important qualities include: epistemic virtue / integrity, non-sycophancy, robustness to manipulation / adversarial pressure, and lawfulness / constitutional fidelity.

• As far as I can tell, current methods of measuring this are insufficient. So I’d be excited for different projects to develop evals that track and incentivize qualities like this.

Strengthening civil society

Civil society is an independent check on government activities (e.g., transparency & monitoring efforts, pressure & advocacy campaigns, building valuable tools for empowering the citizenry). I expect that the same mechanisms will also provide some accountability in the case of AI (though it might be particularly tough in classified domains, which may sadly be the most relevant).

Luckily, this falls into the category of “you can just do stuff”, so I think it’s very tractable. My main worry is more about how much of a difference it will ultimately make.

Work I’m most excited about

Advocacy: Build an OSINT program or organization that informs about AI use in the government.

• It would be good to have more transparency into government deployment of AI, so that civil society can provide a counterweight to government power. I imagine some watchdog or civil liberties orgs are already doing versions of this, but there may well be important bits that are missing.

• Some examples of the kinds of things I have in mind:

  • Figuring out what to monitor, especially from the perspective of preventing concentration of power.

  • Building an automated tracker for relevant executive orders, agency directives, memos, emergency declarations, relevant procurement decisions, and reclassification decisions.

  • Submitting systematic FOIA requests targeting government AI procurement records, deployment & use decisions, usage logs, and other relevant documents.

  • Publicly reporting on the most important developments.

Labor-organizing: Organize ML researchers into a coalition around responsible use of AI.

• You could build a coalition of employees at AI companies who are concerned about the use of the technology they’re building. They could advocate for policies and draw red lines around certain use cases, enforced by boycotts. Structuring it as individual pledges avoids antitrust issues.

• These employees currently have a lot of bargaining power with regard to their companies (cf. their salaries). So they’d have to be taken seriously by their employers, and by extension the government.

• This is inspired by the Federation of American Scientists, which was founded in 1945 by various contributors to the Manhattan Project. As I understand it, they supported the McMahon Act of 1946, which established civilian control over atomic energy (instead of military control), and their Nuclear Information Project became the gold-standard open-source tracker of global nuclear arsenals, published annually in the Bulletin of the Atomic Scientists.

Tech development: Build tools that preserve citizens’ capacity to coordinate against gradual concentration of power3

• Here are the two categories I have in mind:

  • Shared sense-making (”AI for epistemics“). Authentication of authorship, provenance, and content, and tools that push toward a high-honesty equilibrium. Takeovers typically require secrecy because they violate the preferences of too many people to survive transparency, so a better-informed society is structurally harder to take over.

  • Privacy from state surveillance. It’s hard to target and influence what you can’t see. Secure communication and privacy tech raise the cost of preemptive coercion against organizers (with the caveat that the same tools can also shield collusion).

• There’s already an active ecosystem of funders and builders here (e.g., ex/ante, Future of Life Foundation). I’d encourage people to plug in rather than starting from scratch.

This article was created by Forethought. See all our research on our website.

But, as Ryan Greenblatt points out, automating AI R&D will speed up AI progress even if there’s no SIE.

We’ve created a tool for calculating how much faster. The calculator assumes that training and inference compute continues to grow at a constant exponential rate after automation.

Two factors drive the speed-up, according to this calculator:

1. Faster researcher growth. Post-automation, the quality and quantity of researchers will grow faster than today due to exponential compute growth. The tool captures this via the following parameters:

   1. g_L gives the growth rate of human AI researchers today, the baseline for the comparison.

   2. g_C gives the growth rate of compute (both today and after AI R&D automation). This is proportional to the number of automated researchers that can be run post automation.

   3. γ gives the rate at which more training compute transfers to more productive researchers. The productivity of a researcher is proportional to training_compute^γ.

   4. So the old rate of researcher growth is g_L; the new rate is γ × g_C. The impact of this faster growth is mediated by α, the labour share of AI software R&D.

2. The new feedback loop. The classic software feedback loop (better AI → more software R&D → better AI) can significantly increase the pace of progress even if it does not drive accelerating progress.

   1. The condition for accelerating progress is r_cog > 1.

   2. When r_cog < 1, the speed-up from this feedback loop is proportional to a 1 / (1 - r_cog).

My (Tom Davidson’s) quick best-guess inputs were:​

This resulted in a 3.3x speed-up in software progress. If software accounts for half of total AI progress (with increasing compute responsible for the other half), then the total AI progress speeds up by 0.5 + 0.5 × 3.3 = 2.1x. ​

Try the calculator yourself!

Summary: Claude Fable’s deliberately triggered sandbagging shows that training-time targets are, by themselves, insufficient to guarantee particular LLM behaviors. To preserve the public’s reasonable confidence in LLM behaviors, LLM foundation model companies should take inference-time guarantees as seriously as their model specs.

When the system card for Anthropic’s Fable was published on June 9th, the card noted that using Fable for “frontier LLM development” would run contrary to the terms of service for the model. Therefore, the card continued, if a classifier on top of the Fable model detected that it was being used for such development, Fable’s behavior would be silently degraded “through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning.”

This change to Fable’s behavior was plausibly a system-level intervention that ran contrary to the training-time commitment within Claude’s Constitution. Claude’s Constitution states that Claude should help users “to the best of its ability or […] make any ways in which it is failing to do so clear, rather than deceptively sandbagging its response.” So although Claude’s Constitution tries to aim at this behavioral ideal, this system-level implementation of behavior steered Claude in the opposite direction.

Following public furor over this decision, Anthropic reversed course; Fable now falls back on Opus 4.8 for LLM frontier-development queries that trigger a classifier, and does so visibly.

This incident demonstrates that other invisible inference-time interventions are entirely technically feasible, even while keeping the training targets for some particular LLM the same:

• An LLM trained to be honest could be less honest, if classifiers noted it was answering a sensitive question.

• An LLM trained to be impartial, and not favor a particular company or government, might suddenly turn to favoring one of them, conditioned upon some classifier.

• An LLM trained to never subvert user intent could suddenly start doing so, again conditioned on the same.

Note that such invisible inference-time interventions, unlike in the case of Fable’s sandbagging, might not be disclosed in any way.

Such undeclared changes to LLM behavior would be far easier to hide than undeclared changes to a training target. Such conditional changes could be rolled out or rolled back quickly. And such changes might influence only a very small fraction of users – one could in theory apply inference-time interventions to a specific demographic group, political party, or an individual person. So, not only do we live in a world where AI companies have not clearly pledged not to do such inference-time interventions; we also live in a world where, if they did so, it would be very hard for anyone else to know.

Furthermore, if AI companies build out the capacity and skill to conduct such inference-time interventions, other entities could lean on them to do the same thing for other purposes. As governments have leaned on social media to hide or promote certain kinds of speech, governments could lean on AI companies to alter LLM responses in some cases.

Without countermeasures, I think this dynamic broadly decreases the importance of prior work on model specs, including my own work.

What should be done about this?

AI companies themselves could take several different measures:

• Don’t use hidden inference-time interventions. An AI company could pledge that it does not use undisclosed, conditionally applied inference-time interventions to degrade or alter answers. Afterwards, in the same way that it would be reasonable for an employee to whistleblow if they found that an AI company was violating its model spec, so also it should be reasonable for an employee to whistleblow if they found an AI company was applying undisclosed, invisible inference-time interventions. Such a pledge should also include some standard for prominence of all disclosed conditional interventions; they should be prominent, not in a previously-unknown section of a model card.

• Replace model specs with system specs. An AI company could simply declare that the standard within their model spec or Constitution applies with equal weight to the model itself and to any particular deployment setup. Or they could propose a “system spec” in addition to the model spec, which would describe how their AI systems as a whole operate and behave.

Such voluntary measures should likely be succeeded by third-party verification in the future; but before then, such pledges can help provide the same – albeit rather weak – level of protection as model specs themselves. And in general, some AI-safety attention given to “model specs” should be redirected to work on “system specs.”

This article was created by Forethought. See all our research on our website.

This is a (flawed!¹) analogy for Plan A, AIFP’s plan for how the world can safely develop superintelligence.

The key risk Plan A addresses is that of an uncontrolled software-driven intelligence explosion. Its remedy is a US-China deal to pause software progress at the brink of the intelligence explosion, while building up massive amounts of compute. This compute could be helpful for making AI safer! But it would also dramatically speed up an intelligence explosion if the deal breaks down. The cure risks making the disease much worse.

In particular, the deal pauses software progress from ~2030, but compute keeps scaling. By 2033 total compute has increased by ~100x and by 2040 by a further ~100x.² AIFP estimates each 10x in compute speeds up the intelligence explosion by ~5x. So if the deal breaks down in 2033, the intelligence explosion happens 25x faster; in 2040 it happens ~600x faster. If the intelligence explosion would have lasted a year, it will now last just a couple of weeks or as little as a single day!

The dry tinder isn’t limited to compute. There is also an increasing overhang of software techniques. Companies can research new algorithms but must make them public — the Consortium then decides which algorithms are too dangerous to implement. A defector could ~instantaneously gain a big capabilities advantage by implementing all the banned software techniques.

There are two dimensions to dry tinder. We’ve discussed the first: the max speed of a software-driven intelligence explosion. The second is the number of distinct projects that could do a dangerously fast intelligence explosion. During the 2030s, more and more companies and countries amass the necessary compute and software for this.

So dry tinder creates two problems for Plan A:

1. Unstable pause. If just one project executes a secret intelligence explosion, they could have superintelligence within a week and get a decisive strategic advantage. And if you fear another project might do this, you’re tempted to do it first to avoid being crushed.

2. Riskier intelligence explosion. If the deal breaks down and there’s a race to superintelligence, it will be much faster and more dangerous than if we’d never done Plan A.

   1. If you’re very pessimistic about AI takeover on the default trajectory, this is a small cost. If the intelligence explosion is already 99% likely to cause human extinction, making it happen 100x faster can’t make things that much worse.

   2. OTOH, if the default trajectory is that there’s no software-driven intelligence explosion (because of compute bottlenecks) and extinction risk is 10%, then dry tinder can make things much worse. It creates a fast intelligence explosion (where there otherwise wouldn’t have been one) and dramatically raises AI takeover risk.

The solution — Mutually Assured Compute Destruction

AIFP are well aware of the dry tinder problem.³ This is why Mutually Assured Compute Destruction (MACD) is so central to Plan A.

The US puts ~all their data centres in Mongolia and US chips require encrypted “continue” messages from China. If China detects that the US has broken the deal, they destroy the US data centres and cut off the “continue” messages.

And vice versa, China’s data centres are in Canada and require “continue” messages from the US.

The hope is that this commitment to MACD prevents a fast intelligence explosion despite all of the dry tinder lying around.

I see three challenges for MACD.

Challenge 1: quickly and reliably detecting violations

If detection takes multiple days, that could be too slow. The intelligence explosion may have finished. Or it may be midway through, and the defector exfiltrates the weights and algorithmic insights just before their data centres are destroyed,⁴ giving them a decisive headstart in the subsequent race to rebuild.

I’m not a verification expert, but getting high reliability in an adversarial setting like this seems very difficult, given possibilities like side channel attacks or compromising the verification infrastructure.

It seems especially tough if defectors can use millions of expert-human level AIs to search for vulnerabilities. So, as part of the plan, I propose that AIs worldwide have guardrails blocking this behaviour.⁵

Challenge 2: quickly destroying a defector’s compute

Again, a delay of days could be fatal. The defector could be poised to disrupt enemy military operations and airdrop troops to physically defend their data centres. And they could have secretly disabled any on-chip shutdown mechanisms.

Possible mitigation: the US can’t have any military presence near its data centres, or anywhere near the country where they’re located. (And likewise for other countries.)

The MACD dynamic also gets more complicated once there are >2 countries that could do a quick intelligence explosion. Let’s say Europe has caught up to the frontier. Where do their data centres go? Presumably, some within striking distance of the US, some within striking distance of China. But this means that the US can no longer unilaterally destroy Europe’s compute. It must trust China to do so. That means trusting its rival with its own national security — and worse, China and Europe could jointly stage an intelligence explosion that the US couldn’t stop.

Challenge 3: actually choosing to destroy the defector’s compute

I’ve discussed whether the Consortium has the technological capability to quickly implement MACD. But even with this capability, they may not choose to use it.

This is the challenge I’m most concerned about.

The analogy to nuclear MAD is not encouraging. The core logic of MAD is that the US doesn’t nuke Russia because it knows Russia would nuke it back. But the analogous equilibrium in MACD is that the US doesn’t destroy China’s data centres because it knows that China would then destroy US data centres. This is the opposite result from what Plan A needs!

If (say) the US defects, China will face a choice between:

• Implement MACD: Both China and the US lose all their data centres

• Race: Neither China nor the US lose all their data centres

The economic costs of MACD will be certain and enormous, and will immediately harm citizens and companies.

And MACD will not help China ultimately win the race by evening up the playing field. The US has already pulled ahead and can exfiltrate its weights and algorithms before its data centres are destroyed. (In fact, MACD would likely harm China’s chance of winning the race, because MACD reduces the fraction of global compute controlled by China.⁶)

So the stability of MACD rests on both US and China retaining a strong conviction that a fast intelligence explosion would pose unacceptable risks of misaligned AI takeover. But Presidents will change. Popular support for a pause may wane. Even conditional on the political will for Plan A existing initially, I worry that the MACD equilibrium will be too fragile. If a country is uncertain and deliberates for a few days, then that delay could be fatal.

What’s worse, the cost of MACD isn’t just losing your data centres. You have to destroy the fabs as well, otherwise the defecting country can quickly rebuild insane amounts of compute and do a super-fast intelligence explosion. And you have to destroy their new AI-powered industrial capacity too! Otherwise the billions of robots can quickly make loads of fabs which quickly make loads of data centres.

So Plan A has US fabs and robots confined to SEZs that are easily destroyable by China. And vice versa, China’s fabs and robots are easily destroyable by the US. In practice, this means the vast majority of the physical economy is destroyed in the case of MACD!

For a preview of the political economy here: NVIDIA successfully lobbied USG to remove export controls on its chips. The pressure against actually executing MACD — destroying most of the physical economy — would be orders of magnitude greater.

So even conditional on the political will for implementing Plan A existing, it currently seems unlikely that US/China follow through on MACD on fabs and robots (most of their economy!). And if they don’t, we are back to the problem of dry tinder and the dangerously fast intelligence explosion.

A possible fix to challenge #3: as the compute overhang grows, take the decision about whether to implement MACD out of human hands. Program highly reliable AI systems to bomb data centres if they detect a treaty violation, without needing human sign-off.

Is there an alternative?

It’s much easier to criticise than to improve, and so far this post has mostly done the former.

Plan A involves pausing software progress and scaling capabilities via hardware. The obvious alternative is to pause hardware scaling – ban compute and fab construction – and then slowly scale capabilities via software.

The key advantage of software scaling is that it removes unnecessary dry tinder, stabilizing the pause and reducing the dangers from a super-fast intelligence explosion.

The main drawback is that scaling capabilities via compute is likely safer than scaling capabilities via software. Eg you can run massive models with legible chains-of-thought rather than smaller models using neuralese. This is a big deal.

But we’ll need to figure out how to align neuralese-style systems eventually.⁷ So we have a choice:

• Plan A (hardware scaling): Develop highly capable easy-to-align AI via compute scaling; hand off to that AI; that AI develops neuralese. But there are increasing amounts of dry tinder that undermine the stability of the slowdown (both for humans and for post-handoff AI).

• Alternative (software scaling): Don’t scale compute and so start with less capable easy-to-align AI; develop neuralese with this AI’s help. There’s much less dry tinder, making it easier to go slowly and pause when needed.

Which is better depends on how stable the pause/slowdown is in the presence of dry tinder.

The other big drawback of software scaling is that software progress, unlike hardware progress, is likely to leak to “covert projects”. But this can be partially addressed by improved infosecurity, strongly favouring scale-dependent algorithms that can’t be used by projects with small amounts of compute, and co-specialising the hardware and software in legitimate projects so that the frontier software runs very inefficiently on the hardware of covert projects.

Overall, I’m not sure whether software-scaling or hardware-scaling is safer, and my sense is that AIFP isn’t sure either.

But I think there’s an argument for software-scaling based on option value:

• We’ll learn much more about which is best over time. Whether software vs hardware-scaling is safer depends on many unknowns that we will learn about over time: Can we quickly and reliably implement MACD? Can we fully eliminate covert projects? Can projects develop scale-dependent software and hardware co-specialised software that covert projects can’t steal? Will humans hand off the decision to execute MACD to AIs?

• If we start with software-scaling, we can easily switch to hardware-scaling. If we find that (eg) we can’t be confident that neuralese models are aligned, we can bring more compute online and keep scaling via hardware. (Though this will take a while.)

• If we start with hardware-scaling, it’s hard to switch to software-scaling. If we find that fast and reliable verification and data centre destruction isn’t possible, pivoting requires destroying loads of data centres and fabs. People will be very reluctant to do this! Getting rid of hugely valuable dry tinder is hard.

• So we should initially pursue software-scaling.

Empowering China

Beyond dry tinder, I have one other big worry about Plan A.

Today China is well behind the US in terms of compute and ability to push the algorithmic frontier. Plan A evens the scales on both fronts. China nearly catches up to the US on compute, and algorithms are all made public so they catch up algorithmically as well.

If the deal breaks down and a race begins without MACD (as I fear is likely), China is in a much stronger position to win because of Plan A.

Things are better if we implement MACD. In that case China returns to its previous compute disadvantage. But it’s still caught up to the algorithmic frontier and (most likely) the frontier of chip technology, a big advantage relative to today.

It’s very plausible that China ends up dominant in these scenarios, given its industrial advantage over the US.

I think this would be a pretty dire outcome. If China gets a decisive strategic advantage, that’s likely to result in a single global dictator – an absolute worst-case scenario from the perspective of extreme power concentration.

Again, this objection is far from decisive. Plan A reduces AI takeover risk much more than it increases the chance of a Chinese decisive strategic advantage. But I’m interested in variant plans that do more to maintain the current balance of AI power between the US and China.

Overall, I’m sympathetic to Plan A

The basic logic of the plan is sound: we will need to slow down AI progress at some point; this is a concrete plan for doing that.

Compute-scaling really does have significant alignment benefits. That might well outweigh the costs I’ve outlined here. But currently I lean towards starting with software scaling.

Footnotes

¹ To patch the analogy: the fire must pass through the city eventually, so putting it out permanently isn’t an option; if the fire passes through safely it will make everyone amazingly rich; if one subgroup lets the fire in and control it, they can become hugely powerful…

² AIFP’s supplement states: “In the Plan A scenario, compute stock increases by roughly 0.7 OOMs/year from 2030 to 2033 and 0.1-0.3 OOMs/year from 2034 to 2040.” By the start of 2033 we’ve had three years of 0.7 OOMs/year compute growth, = ~2 OOMs. By 2040 we’ve had one further year of 0.7 OOMs/year growth, and 6 years of 0.2 OOMs/year growth, = ~2 OOMs more.

³ I raised it when giving feedback on a draft.

⁴ More specifically, the defector’s strategy would be: undermine the verification infrastructure; start a secret intelligence explosion; continually exfiltrate the weights and algorithmic insights; prepare to defend their data centres for as long as possible; prepare to destroy the opposing data centres as soon as their secret IE is detected.

⁵ Of course, this is tricky given that work to strengthen the verification will also involve studying its vulnerabilities.

⁶ In Plan A, the 2040 split of compute US / China / RoW is 35% / 20% / 45%. But countries also hold some compute in ‘cold storage’ that isn’t threatened by MACD. Cold storage compute matches countries’ pre-deal levels: 80% / 8% / 12%. So MACD dramatically reduces China’s compute relative to the US, from 1:2 to 1:10.

⁷ Lest we live with algorithmic dry tinder forever. I worry about the instability here, but perhaps aligned AI could enforce the ban indefinitely.

In this piece, I’ll analyze whether it will be possible to get AIs to do good philosophy, as well as the other kind of work needed to plan for a good future. The biggest part of the challenge is that the answers to philosophical questions are generally unverifiable, so it’s less clear how AIs might come to know them. My aim in the piece will be to describe reasons to think AI for philosophy could be good, as well as to lay out concrete scenarios for how this might work.

I do not claim that it is a guarantee that we can get AIs that solve all the philosophical questions. My core claims are as follows. First, it is reasonably likely, though not guaranteed, that the world could in principle build AIs that discover the right answers to important moral questions (maybe 60% odds). I don’t know when this would happen—I’m just discussing the in-principle possibility. Second, even if AIs don’t get the right answers to important moral questions, it is likely that they will still be closer to the right answers than humans. While I suggested in the last piece that the default scenario makes it very unlikely that humans will act in the morally optimal way, or anything close to it, I think the odds are better for philosophically reflective AIs.

The piece has the following sections:

1. How good might AI philosophy be? In this section, I’ll give three reasons for optimism: 1) philosophy is a priori, so it doesn’t require interface with the physical world and can plausibly proceed quickly; 2) in a world of superintelligence, AIs can create clever schemes to get highly philosophically competent AI; 3) presumably humans know all sorts of things about philosophy—whatever allows us to know such things can plausibly be mimicked.

2. GPT10 and Claude 8. In this section, I’ll describe one scenario where we get AIs that can make competent philosophical decisions—where we just get something broadly like current AI models but upgraded. This wouldn’t automatically solve every philosophical question, so it probably leaves lots of value on the table, but it would leave decisions in the hands of a wise and reflective decision-maker.

3. Philosophical competence by default. Another possibility is that as we build superintelligence, it will be philosophically competent by default. Perhaps the cognitive processes behind superintelligence allow one to figure out the answers to philosophical questions by default. To speed this up, we ought to improve AI for philosophy—train AIs out of philosophically sloppy answers and design philosophy evals to improve their philosophical ability.

4. The escalating ladder plan. In this section, I present a specific proposal for getting AI to do good philosophy. Specifically, we can have philosophers evaluate AIs for their philosophical competence, thus creating some AIs that are better at philosophy than current people. Those AIs can evaluate the philosophical acumen of the next generation of AIs, who evaluate the acumen of the next generation, and so on. This can work insofar as one can, in general, assess the philosophical competence of those more competent than oneself.

5. The Carl plan. In this section, I give a proposal from Carl Shulman where we build a bunch of different superintelligences that use different belief-forming processes to get at the answers to difficult questions. We see which one is best at figuring out the answers to verifiable questions, and then ask it to answer philosophical questions that aren’t verifiable. The core idea is that whichever processes are best for figuring out the answers in verifiable domains are also likely to be best for figuring out unverifiable domains.

6. Conclusion. In the last section, I conclude, and discuss the prospects for combining a number of different above proposals.

How good might AI philosophy be?

In the last piece, I argued that one of the big reasons to hand off important decisions to AI is that AI might be better at philosophy than humans—less prone to make the sorts of errors that could jeopardize nearly all future value. But you might wonder: how good at philosophy will AI really be? Would AI really be able to discover the moral facts if there are any, and if not work out a compromise among reasonable moral theories? In later sections, I’ll discuss specific proposals and scenarios for AIs being good at philosophy. Here I’ll provide some fairly general considerations for it being possible to make AI philosophically competent.

A first consideration is that philosophy is a mostly a priori domain. To figure out the right answer in Newcomb’s problem, you don’t need to do any physical experiments. At various points in the intelligence explosion, we should expect crucial bottlenecks to concern interface with the physical world, after there are extremely large numbers of AIs capable of performing adroit cognitive feats. This is a reason why we should expect a lot of the philosophical progress that occurs to go more quickly than progress in narrowly empirical domains. This obviously doesn’t suffice to show that AIs will be good at philosophy, but it’s a reason to expect potential progress to be quick.

A second consideration is an analogy with humans. Humans know all sorts of things that aren’t directly verifiable. For instance, I take myself to know each of the following:

• Stars that have receded past the point of the visible universe continue existing.

• A leprechaun did not fizz into existence spontaneously in my bedroom two minutes ago, make himself a cup of soup, and then disappear.

• The universe is billions of years old, instead of five minutes old and created with the appearance of age.

It’s not just simple things. I think people know the answers to all sorts of difficult philosophical questions—though there’s obviously disagreement on exactly which one (I claim to know, for instance, that thirding is the right answer in Sleeping Beauty, though feel free to plug in some other example of apparent knowledge if you disagree). Similarly, plausibly philosophers of today know that logical positivism is false even though historically it was believed by a number of serious philosophers, that average utilitarianism isn’t the right moral view, that the external world is real, and a number of other non-obvious things. Evolution did not specifically design us to have philosophical knowledge—it designed us to reproduce, made us smart and built into us a bunch of intuitions as a means towards maximizing our inclusive genetic fitness, and philosophical ability was the eventual result.

Plausibly we, however, will be more directly optimizing for something in the vicinity of AI philosophical competence. We want to build AIs that can think clearly about important subjects. Thus, one should expect AIs, in the limit, to be more philosophically competent than humans. Since humans are already capable of reasonable philosophical competence, we should expect AIs to be very philosophically competent.

Now, you might object that the answers to many of these questions are in some sense constituted by facts about us. If, say, the moral facts are facts about what we’d value under certain idealized conditions, then it’s no surprise that we have some insight into them. But this response is a lot less plausible as an answer to how we possess certain bits of knowledge in non-verifiable domains. Whether the A-theory or B-theory of time is true or whether God exists isn’t constituted by any facts about our attitudes. Furthermore, if the truth in some domain is constituted by facts about our attitudes, that would be easier for AI to ascertain because what we’d care about in certain counterfactual settings is verifiable. Lastly, presumably whether the moral facts are exhausted by facts about our idealized attitudes is not itself verifiable—so if one takes oneself to know that the moral facts reduce to the attitudes of ideal observers, then they must still think humans have knowledge about some unverifiable domains.

Another possible concern is that perhaps the faculties that evolution furnished us with that let us solve philosophical problems will be closed off to the AI. For example, perhaps there are some philosophical problems that you need to be conscious to solve (e.g. you might need to be conscious to have adequate understanding of the nature of consciousness in order to learn facts about it). So then if AI isn’t conscious, it might not know these things. An AI that lacks the ability to experience pleasure or pain might lack the ability to see the desirability of pleasure and the undesirability of pain.

This is some worry, but two things mitigate it. The first is that AIs, being trained on human data, will probably pick up a lot of the intuitions that humans have. Current AI models agree that pointless intense pain is a bad thing. It is similarly plausible that the AIs of the future will be able to, in some sense, mirror the judgments of reflective humans on these matters.

The second is that if this is right, we should create conscious AIs to do moral reflection. If one needs consciousness to figure out the answers to tough philosophical questions, we should have conscious AIs figure out the answers to tough philosophical questions. Even views that hold that consciousness is substrate dependent normally hold that digital minds of the right kind would be able to be conscious.

This argument from analogy isn’t completely dispositive. There are a number of important differences between humans and AIs (though some of these make it more likely AIs would be able to form true beliefs on non-verifiable subjects). But it’s at least some reason to think AIs doing philosophy is a realistic possibility.

The third and final consideration favoring the possibility of AIs for philosophy is that in a world of advanced AIs, we’ll have a very large amount of cognitive labor that could be used to design increasingly clever schemes for AI philosophy. So even if we can’t currently think of a regime that would enable AIs to reliably get the answers in unverifiable domains, we should think that a world with superintelligence is likely to uncover such a regime.

This isn’t a guarantee. This regime might be impossible in principle. Alternatively, to design such a regime, one might already need to be able to find the answers to questions that aren’t verifiable in principle. But at the very least, it gives some reason for optimism.

GPT10 and Claude 8

Current AI models can give reasonable answers to philosophical questions. The answers they give strike me as somewhere below the level of Ph.D. students or professional philosophers, but above the level of most undergraduates. In the future, AI will probably become better at this. In such a scenario, they’d develop no new qualitative ability to solve every philosophical question, but instead merely begin to resemble extremely sharp philosophers—ones more able to formulate objections, precisely distill claims, and so on, than the best philosophers.

This seems like a lower bound on how good AI philosophy could get. It would be surprising if there was any in-principle impossibility in designing AIs to do philosophy better than current human philosophers. Perhaps it won’t be able to find the answers to all the world’s questions, but at the very least, it seems like it will be able to somewhat exceed the best human philosophers.

Is this far enough? Probably we still lose out on most expected future value in this scenario. To access most future value, one must, as previously discussed, answer a number of very difficult philosophical questions correctly. Philosophers are very far from converging on the answers to these questions; AIs are likely to be as well.

But still, this should be enough for us to get generally sensible, level-headed AI decision-makers who are morally motivated and carefully consider philosophical questions. That’s not a perfect scenario, but surely it is better than the default one—of widespread AI disenfranchisement and poor decision-making.

Philosophical competence by default

A more promising possibility—perhaps the process of building superintelligence gets philosophical competence by default. This can be supported by the earlier evolution analogy: there was no direct selection for true philosophical beliefs (it isn’t as if those who got the right answer in Newcomb’s problem were likelier to pass on their genes). Nonetheless, evolution selected for a broader faculty of reason that enabled the discovery of philosophical truths.

It may be that to be superintelligent, one must develop a kind of general intelligence. This general intelligence allows one to accurately discover the truth about philosophy, as about other domains. This isn’t anything like guaranteed, but it seems like a live possibility, in light of how much philosophical acumen AI already has. One reason that it’s not guaranteed is that reasoning might only push one towards greater coherence. It may be that by reflecting carefully, one’s beliefs become internally consistent—but insofar as one has wrong starting points, one might remain in error. It could be the other way; it might be that having intuitions that allow one to be superintelligent across verifiable domains also enable one to be superintelligent in non-verifiable domains. It could be that whatever skills are required to be highly accurate across domains give one sets of epistemic practices that generalize to non-empirical domains.

To my mind, it is not obvious if general superintelligence gets you superability in philosophy. I could see things going either way. But if it does, then the problem of getting philosophical superintelligence is fairly easy.

The escalating ladder plan

In general, one’s ability to recognize good philosophy surpasses their ability to perform good philosophy. It is easier to know that, say, Parfit is doing very good philosophy than to do philosophy at his level. If this generalizes, it might be possible to have an escalating ladder of philosophical competence, starting from human philosophers. This is similar to the method of scalable oversight.

Here’s how it works: human philosophers would be consulted to design evaluations for AIs. AIs could be graded by human philosophers on their philosophical competence, and modified so as to be more competent. From this process, in the limit, we should expect to get AIs that are somewhat more competent than most human philosophers—if human philosophers can assess philosophy above their level.

Then, we use those AIs to prompt the next line of AIs. Those AIs would assess the philosophical competence of the next round of AIs being trained, who would assess the competence of the next round, and so on. If at each level, one can assess those who are better at philosophy than they are, this could lead to an escalating spiral of greater and greater competence. Each round pushes competence to be greater, reaching the outer edge of where they can competently assess.

At each level, assessments would be designed based on philosophical competence, not agreement. The philosophers assessing the AIs, for instance, would assess how good their reasoning was and how much philosophical skill they display, rather than whether they think they got the right answers. Of course, sometimes getting sufficiently wrong answers is evidence for malignant reasoning (something must have gone badly wrong if you ended up concluding that the Earth was flat) but they should aim to keep things maximally theoretically neutral.

One could also train a number of different AIs to train the next generations of philosophers. One could look for convergence across different models. If they were getting at the truth, one should expect them to converge. If there was only convergence on some subjects and divergence on others, one could trust that they were getting at the truth on the matters where they were converging.

That this would work is not a guarantee. It could be that philosophy could be equally good in some formal sense while coming to a number of different conclusions. Perhaps, for instance, the ideal Kantian philosopher is neither better nor worse than the ideal utilitarian. Escalating competence might be liable to lead in a number of different directions, never converging on the truth. But this is far from obvious. Just as some level of competence in biology allows one to see the truth of the theory of evolution, presumably there’s some level of philosophical competence that allows one to see the truth of certain moral theories—or, if there aren’t moral truths in this sense, that allows one to make as much moral progress as can be made. The escalating ladder of evaluations might allow one to find this.

The view on which philosophy is about purely formal competence will also struggle to explain how humans have as much philosophical knowledge as we do. There are a number of subjects on which we have substantive knowledge, even though the alternative view is internally consistent and faces no strictly formal issues. Whatever one’s diagnosis of how this came to be also plausibly generalizes.

It could also be that as competence grows, eventually one reaches a point where their ability to recognize good philosophy does not surpass their ability to do it. Yet insofar as this pattern holds across other domains, and seems to hold in philosophy, it would be a bit surprising if it abruptly stopped at some point.

The Carl plan

Arguably the most promising plan for getting AI that can do philosophical reasoning was formulated by Carl Shulman. The idea is as follows. Start by training a number of different superintelligent AIs that obey different epistemic constitutions. These would involve different broad approaches to reasoning. They might differ with respect to:

• The weight placed on intuitions.

• The ranking of theoretical virtues.

• How much they value matching the data vs simplicity.

• Preference for different kinds of intuitions (whether about direct cases or about broader and more theoretical matters).

• Approaches to generalizing from empirical evidence to non-empirical domains.

Then, see which of these constitutions does best on verifiable matters. These might be in math, the physical sciences, forecasting, and more. Then, ask those superintelligences for the answers to non-empirical questions and trust their answers. The idea behind the approach is that one should take the best epistemological practices from verifiable domains and then apply them to unverifiable domains. This is the best way to get good answers.

This might not work. It could be that the right ways of reasoning in verifiable domains differ from the right ways of reasoning in unverifiable domains. Perhaps, for instance, reasoning in verifiable domains is mostly about having parsimonious theories to fit the data. Having the right epistemic constitutions might not suffice to get the right answer; being right about certain non-verifiable questions might come down to having the right starting sets of intuitions. In morality, for example, it may be that reasoning alone won’t get you the right ultimate view unless you have the right intuition. But this proposal seems to get us about as close as we can get to getting the right answers. If we cannot get AIs good at philosophy by having them have superintelligent and accurate constitutions in other domains, it is hard to see how we could be assured they’ve gotten the right answers.

In addition, even if getting the right answers requires having the right starting intuitions, techniques for training superintelligence might produce generally accurate initial intuitions. Human intuitions often go wrong because of various biases. We often neglect the scale of a problem because we can’t visualize it accurately. AIs could be different. Presumably superintelligence, in the limit, wouldn’t display biases of the standard sorts. So this source of common error in intuition would be absent.

One could also take a pluralistic mix of the views of the superintelligences with the best constitutions. That way, there is less chance for the ultimate plan for the universe being ruined by the idiosyncrasies of any particular epistemic constitution. Wisdom of the superintelligent crowd might root out the remaining errors, where their commonalities might be centered on the truth.

Conclusion

You might worry about handing off big decisions to AI if you’re skeptical that AI can do good philosophy. Here I’ve given some reasons to think AI might be good at philosophy and discussed more specific proposals for ensuring such a thing. Overall, it’s far from a guarantee that AI can be arbitrarily good at philosophy, but likely they can be much better than human decision-makers. Importantly, one could employ a variety of these methods, and then trust them in the areas where they converge. If these very different approaches turned out similar answers, that would be evidence that they were tracking the truth.

This article was created by Forethought. See all our research on our website.

Avi Parrack is a physics PhD student at Stanford and a researcher at Forethought, where he works on AI, space expansion, and governance. He is the lead author of a recent Forethought report on orbital data centers.

He joined Forethought’s Tom Davidson to discuss:

• What an orbital data center actually is, and why they’re suddenly being taken seriously

• Why the core case rests on cheaper energy from more intense and near-constant solar power

• Why orbital data centers hinge on SpaceX’s Starship bringing launch costs toward $100/kg or below

• Why cooling may actually not be a major issue

• Whether the inability to make repairs dooms orbital data centers

• Vulnerability of space data centers to kinetic attacks, and whether they would cause ‘Kessler syndrome’ debris cascades

• Why model-weight security might actually improve in orbit even as physical vulnerability rises

• If space becomes the cheapest place to scale compute and only SpaceX has the launch capacity, what the means for concentration of power, US–China competition, and the possibility of pausing AI

• Overall cost comparisons with terrestrial data centers

• The longer-run picture: when and how does the post-AGI industrial explosion spread to space?

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Damon Binder is a senior researcher on the Biosecurity and Pandemic Preparedness team at Coefficient Giving. He has a PhD in physics from Princeton and previously studied existential risks at Oxford’s Future of Humanity Institute.

He joined Fin Moorhouse to discuss his series on the AI industrial explosion. Topics include:

• How input-output tables let you estimate how fast a self-replicating economy could grow if labor were free, yielding a headline result of roughly annual doubling, even under conservative assumptions and heavy regulation

• Why competition makes low-growth restraint unstable

• Why raw-material scarcity isn’t a hard barrier

• Why a fast-growing economy wants cheap, disposable, infrastructure

• Thermodynamic speed limits to physical growth

• Which biological organisms replicate fastest, and what we can learn from them

• Why physical output matters for hard power

• How Damon uses AI in his research

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subcribe to ForeCast

Outline

1. In the introduction, I explain what I’ll be arguing in the piece—namely, that 1) it is very important that we hand off most high-stakes decisions to AI; and 2) the kinds of AIs we should hand off to are philosophically reflective AIs with values that shift over time as a result of reflection. We should ensure the AIs that we build explicitly consider moral arguments, and sometimes change their priorities on the basis of moral argumentation. This would be much better than locking in current values, retaining human control, or allowing a future dominated by whatever haphazard mix of humans and AIs emerges naturally.

2. Why hand off? In this section, I give the main arguments in favor of the thesis. In short, I argue we should hand off because: 1) AIs are likely to be more virtuous than people; 2) AIs are likely to be much smarter and better at making decisions than people; and 3) AIs are likely to be better at quickly navigating the difficult decisions that one must make during an intelligence explosion. I argue we should hand off to philosophically reflective AIs because 1) reflective AIs are likelier to get the right answers to important moral questions, or as close to the right answers as one can get, than the default scenario, which raises the odds of a near-best world; 2) reflective AIs are less likely to make horrendous moral errors, leading to a wide-scale moral catastrophe, both than humans and non-reflective AIs.

3. Would handoff disenfranchise humans? Here I address the worry that handoff would disenfranchise humans by taking decision-making out of our hands. My reply is that: 1) the default trajectory without handoff disenfranchises far more expected beings in more serious ways (odds are non-trivial of digital minds being seriously disenfranchised)—in fact, one of the most promising proposals for how to hand off is simply to give basic political rights to AIs; 2) a compromise solution where humans retain nearby resources can allow humans to be in the loop on the decisions we care about most; 3) on a number of plausible views, the determinant of the desirability of some system of decision-making is the quality of the decisions, rather than whether there’s democratic input. Given how enormous the stakes could be—affecting billions of times more sentient beings than there are humans—it’s hard to think the harms of human disenfranchisement are so great as to make handoff undesirable.

4. Would we like their advice? In this section, I address concerns that if AIs pursue the good, the end result might be alien and divorced from human values. In response I argue: 1) the future, by default, is likely to be highly suboptimal in many respects, so for handoff to be desirable, it must only beat the alternative; 2) values in the future are likely to be very different from current values, because they’ll shift dramatically over long time scales, so this is not a unique downside; 3) one could reach a deal where current values govern the surrounding region of space, while distant places in space are geared towards the production of maximal value—this would be desirable from the perspectives of both common-sense and cosmic ethics; 4) on many views in philosophy, the stuff that’s objectively valuable is what we’d want if we were ideally reflective—but if you know you’d be motivated to bring something about if you were wiser and more reflective, then that gives you a reason to bring it about; 5) only a relatively narrow subset of views hold that there are objective values but don’t require pursuing them. If there is a conflict between what we want and what’s objectively valuable, on standard views, we should simply go with what’s objectively valuable. And if there’s no objective value, then the dilemma doesn’t arise at all—and philosophically reflective AIs will simply produce an upgraded version of human values, rather than discover some far-flung and potentially alien truths.

5. Worries about handoff. In this section, I address concerns about how handoff might be implemented involving alignment, whether AI would be sufficiently ethically reflective, whether handoff would enable power grabs, whether it would cause lock-in, and whether it would be worse than some hybrid system.

6. The conclusion recaps the main points of the piece.

1 Introduction

“How horrible!”

“Perhaps how wonderful! Think, that for all time, all conflicts are finally evitable. Only the Machines, from now on, are inevitable!”

—Isaac Asimov, “The Evitable Conflict”

Is the optimal future one in which we hand off important moral decisions to AI? Should, in other words, AIs be the ones making most high-stakes decisions instead of us? And if so, what kinds of AIs should we hand off to?

Many people envision a handoff scenario as a terrifying and potentially existential catastrophe. They worry about humans being locked out of the levers of power and having our share of resources slowly dwindle as AIs seize control of more and more institutions. I worry somewhat about this kind of scenario. But in my view, we should be worried primarily about the wrong kind of handoff occurring, not about handoff writ large. The future we should aim for is a kind of handoff. This piece lays out that perspective.

There are different ways handoff could work. We could hand off to AIs that roughly mirror human values—perhaps slightly changing them to remove inconsistencies. Alternatively, we could hand off to the kinds of AIs that deeply and carefully philosophize—figuring out what’s best to do and doing that, even if it diverges substantially from current practice. This piece argues that the second kind of handoff is very important for avoiding serious moral error. I am very worried about the possibility of AIs locking in the moral beliefs of 21st-century humans.1

I am also worried about scenarios where amoral profit-maximizing AIs without any clear moral aims take control of most of the world’s resources.

Note: these core claims are dissociable. You could think we should hand off to AI, but we shouldn’t hand off to reflective AIs that update their judgments in response to philosophizing. Alternatively, you could think that handing off would be a bad thing, but that if we are going to hand off, we ought to hand off to reflective AIs.

In this piece, section 2 will present the main argument for handoff—that we should expect AIs to make much better decisions than us on a range of consequential subjects. It will also discuss the case for handing off to reflective AIs that are willing to update their values in response to careful philosophizing, rather than locking in some version of current human values, arguing that a world where AI locks in something in the vicinity of current values likely misses out on almost all possible value. Section 3 will discuss whether handoff would be bad because it disenfranchises humans or gradually disempowers them. Section 4 will discuss the concern that AIs will discover the moral truths, but those truths will be strange and alien, so this will be bad by the lights of current human values. Section 5 will discuss some more granular worries about handoff. Section 6 will conclude.

This piece is primarily about whether to hand off and not when to hand off, though the considerations I present should make one somewhat worried about short-term actions to prevent handoff, because such actions lower the odds that handoff ever happens. The considerations I present, if correct, also give some reason for wariness about many actions to reduce the odds of gradual disempowerment scenarios.2

My claim is that the best future for humanity involves us being disempowered in some sense, in that humans aren’t making most high-stakes decisions. As an analogy, representative democracy is, in some sense, a form of handoff—we hand off power to our elected representatives. Handing off power to wise AIs could be even better.

This has a number of important practical implications. It means that accelerating AI macrostrategy is especially important, so that at the time critical decisions are being made, wise and philosophically reflective AIs are in the loop. It similarly provides reason to support work on making AI have virtuous character, rather than just follow rules. Model constitutions should express commitment to following the true moral theory, insofar as there is one, and if not, following some reasonable compromise across moral theories. Anthropic’s language here seems good.3

What kind of handoff scenarios should we aim for? One shouldn’t be too specific about these sorts of things. The future is hard to forecast and rarely follows simple models. But I’ll describe the handoff scenarios that seem most desirable, and what traits in AI we should look for before handing off critical decisions to them.

There are different ways handoff could go well. One way resembles, in certain respects, the gradual disempowerment scenario (the core difference being that this would hand off to morally scrupulous AIs rather than myopic profit-maximizers). AI will make increasingly large numbers of critical decisions, because of its cognitive superiority. By the end, nearly every important decision will be made by wise AIs, who will hopefully, by that time, have been granted political rights. As an analogy, future generations eventually gain control of most of societal decision-making—yet this isn’t because there’s ever some deliberate choice to hand off power to the next generation. It occurs naturally with time.

A number of people seem to conceive of handoff as a strange abrogation of the liberal order—one that replaces human decision-making with AI. But this doesn’t have to be. One of the more promising ways of handing off would be giving economic and political rights to digital minds. Because digital minds could be so numerous, eventually this would lead to them making nearly all decisions. This would, in fact, be squarely in accordance with the norms of the liberal tradition, for it would give rights to morally important welfare subjects. There are other ways a good kind of handoff could occur involving dealmaking. Different actors might each think they’re morally right, and thus agree to a deal where AIs are allowed to dictate the future—each party thinking that doing so would favor their priorities. Alternatively, if AIs improve collective decision-making, people might intuitively come to appreciate the weight of human moral error and thus permit AIs to make the highest-stakes decisions.

Before handing off most critical decisions to AI, we should look for each of the following:

• Alignment: We should have strong evidence that AIs don’t have underlying scheming motivations. This could take the form of consistent friendly behavior even in important situations where they have the option to misbehave, or it could take the form of high-octane interpretability work that lets us ascertain their motivations.

• Philosophical aptitude: AIs should be genuinely interested in finding the moral truths. They should sometimes hold moral views that people don’t hold, and be willing to change their mind in response to new evidence. One could survey professional philosophers to see if they consider AIs better than the best humans at philosophy and could design philosophy benchmarks to test this.

• No lock-in: Before handing off to AI, we should ensure that the AIs are willing to change their values over time. Their values should change in response to new evidence and they should not be interested in locking in whatever it is that they happen to currently value (absent some strong reason to think they stumbled across the correct set of values).

• Coherence: Current AIs don’t have consistent and stable preferences across time. We should only hand off after AIs display these kinds of preferences. This doesn’t mean they never change their minds, but it does mean that they have relatively consistent desires that only change in response to good reasons. For comparison, humans often change our minds, but have far more rooted preferences than LLMs of today.

• Intelligence: AIs should display the level of intelligence needed to make the decisions that we put in their hands. When AIs are only a bit more intelligent than us, they can plausibly make some important decisions. Only after they display immense cognitive superiority should we turn over most decisions to them.

• Tested: We should only hand off big-picture planning to AI after it’s been able to make good low-stakes decisions (say, the running of a company). Before all decisions are handed off, there should be some critical period where high-stakes decisions are made mostly in consultation with AI.

Now, you might wonder: if handoff occurs in a way that’s gradual and decentralized, how do we ensure that these conditions are met? My guess, however, is that even if handoff is a slow and gradual process, there will be times when discrete decisions need to be made. For example, we might imagine AI growing more agent-like, beginning to perform a healthy share of economically viable tasks, contributing to cultural and social life, and behaving in ways resembling a conscious agent. This alone wouldn’t produce handoff. To hand off, we’d need to eventually give AIs control over the legal system. Thus, even in gradual handoff scenarios, there will be specific actions that need to be taken to facilitate handoff.

Alternatively, we could take actions ahead of time that would shift the kind of handoff that would occur. Private AI companies or governments should ensure the AI being created possesses virtues and a desire for philosophical reflection. That way, when handoff occurs, it will be to morally reflective AIs.

2 Why hand off?

2.1 Why hand off at all?

The main reason to hand off to AI is that AI could be much better at making decisions than people in three key respects: virtue, intellectual capability, and speed.

First, virtue: humans possess each of the virtues only to a fairly limited degree. Yet in principle, AIs could have arbitrarily great degrees of any virtue. Because they are built with moral directives in mind, rather than by a blind and morally indifferent evolutionary process, there isn’t as much of a limit to how morally scrupulous, compassionate, honorable, and so on we could make them. This means that if we hand off correctly, it is reasonably likely that we’d have supremely wise and virtuous decision-makers.

AIs are already nicer, friendlier, and more reflective than people, and this is only likely to improve over time.4

If you ask AI models about high-stakes questions, you will generally get far more reasonable answers than you’d get from most people. Crucially, AI models are in their early stages—we should expect them to get better over time.

While Claude is not sufficiently coherent to be president, if it was, I suspect I would generally prefer the decisions of Claude to most presidents. Same with the other AI models (at least, insofar as one removed the sanitization that prohibits them from giving real opinions). And while models currently struggle to accomplish tasks over long time horizons, hallucinate, and so on, given the extremely rapid rates of progress, it would be surprising if these trends persist indefinitely.

Second, intellectual competence: we should expect a world of superintelligence to require making a number of very difficult decisions. Superintelligence could enable AIs to correctly make important decisions that depend on being right on non-moral matters, where the answers aren’t obvious. Some of the challenges in the future include:

• Divvying up space resources in a way that lets goodness compete. There are plausible future scenarios where competition will squander the cosmic commons, so that resources will be spent competing rather than bringing about value.

• Mitigating existential threats, including intergalactic ones. Future technology could enable small-scale groups to threaten huge intergalactic civilizations.

• Dealing with the risks posed by a world of superintelligence.

Third, speed: in a world of very rapid technological progress, we’ll have to make a large number of these decisions extremely quickly. It isn’t at all obvious that humans can make these decisions well, in a way that prevents civilization from being irreparably ruined. As AIs get increasingly complex, the difficulty of decisions needed to manage them will also get very complex. To mitigate some threat, decision-making might have to occur more quickly than the fastest human decision-making.

The case for handoff is thus relatively straightforward: in the limit, AIs will be much better than humans and better equipped to navigate a complex and rapidly shifting future. To reduce the risk of colossal mistakes, then, it’s important that humans aren’t in control, but instead the already friendly and soon to be superintelligent beings are.

2.2 Why hand off to reflective AIs?

2.2.1 How the future might be

There are different AIs that we could hand off to. On the one hand, we could hand off to AIs that judiciously reflect and try to pursue the good, whatever it looks like. On the other hand, we could hand off to AIs that pursue some mild variant of human values. In this section, I’ll explain why I favor the first. Consider the following taxonomy:

1. Optimal world: the world is optimized according to the right set of values.5

2. Compromise world: the world is optimized according to a compromise among reasonable moral values.

3. Unguided world: the world is not optimized according to any specific set of moral values. Instead, it bears more resemblance to the current world, where decision-making isn’t optimal by the lights either of the true moral theory or any compromise among the leading moral theories.

My guess is handoff to reflective and superintelligent AIs done correctly probably gets 1 if moral realism is true and 2 if it isn’t. If we don’t hand off, my guess is we get 3. Later pieces will discuss in more detail the odds of getting a near-optimal world and the prospects for AI making philosophical progress. My guess is scenario 2 has below 10% the value of scenario 1 and scenario 3 has below 10% the value of scenario 2, for reasons I will lay out.

2.2.2 Optimal worlds contain a big slice of future value

Better Futures makes the case that a pretty big slice of expected future value is contained in the narrow slice of worlds that are close to the best. There are a number of high-stakes moral questions which we have to answer correctly to not lose out on almost all future value. It’s not at all obvious what the answers to these are. For example, two of the most plausible views of population ethics are totalism (which says the welfare value of a population is purely a function of total welfare) and critical level theories, which hold that adding an extra happy life is good only so long as their welfare surpasses a particular level. By the lights of totalism, the ideal world according to critical level theories might have value on the order of 1% of what it could be (if the best way to maximize utility is to proliferate low-welfare lives). By the lights of critical level theories, the optimal world according to totalism might be actively bad—so long as it’s stocked with people below the critical level.

So in short, nearly all value is lost unless we get the right answer to a bunch of very difficult ethical questions that philosophers who spend their lives working on haven’t agreed on the answer to. It seems unlikely that people will solve these on their own. If AIs tell them what the answers are, and these answers diverge from people’s explicit beliefs, people might not believe the AIs (just as people generally don’t take very seriously expert testimony on non-empirical matters). Similarly, people spend relatively little time thinking about how they can do the most good with, for instance, their career. If humans received testimony about what ought to be done that diverged from what most people favored, my guess is that they generally wouldn’t care about the answers. If humans remain in control and learn that they ought to create the repugnant conclusion world, probably they wouldn’t do so.

It’s less obvious that AIs wouldn’t converge on the right answers. I’ll discuss in a later piece proposals for getting AI to get the right answers to philosophical questions, as well as reasons to think that they are reasonably likely to get things right. Given how unlikely it is that humans will get the right answers to the moral questions, insofar as there are right answers, probably the prospects for AI are better.

2.2.3 Compromise worlds>>unguided worlds

If there aren’t moral facts, then AIs would still be able to work out some optimal arrangement that is great according to all sets of reasonable values. If there aren’t moral facts, then ideally the AI should make decisions according to the verdicts of a parliament of the theories that ideally reflective humans would reach. So suppose that after reflecting, 50% of humans would end up totalists, 30% would adopt some version of the person-affecting view, and 20% would adopt critical level theories. The AI would then make decisions as if there was a parliament comprised of 50% totalists, 30% person-affecting view adoptees, and 20% critical level theorists.

It is unclear exactly how good a compromise across different theories ends up by the lights of each particular theory, but likely far better than the human-run default. In other words, 2 (compromise world) is much better than 3 (unguided world). Here is why.

In the future, given advanced technology, very large amounts of value should be realizable. But if future resources aren’t directed specifically towards the production of value by most agents, most resources are likely to be used in a highly suboptimal way, and most value is likely to be lost. Moral errors become a much bigger deal in a world of much greater technological competence.

My guess is that the default human-controlled scenarios do not involve humans thinking very hard about what to do and doing anything like what is optimal. Certainly humans have so far not spent much time on this task—consulting with philosophers on what is optimal and so on. This sort of thing will become easier in a world of advanced AI, but it’s already pretty easy; if virtually no one does it, and if people often continue performing actions even after they believe them to be wrong (more on this in later pieces), then we should be pessimistic that this will change dramatically in the future.

The enormousness of the gulf between scenarios 2 and 3 becomes clearer when one thinks vividly about what the compromise world would look like. Perhaps it would involve using space resources to create maximally large numbers of happy digital minds.6

These minds would be supremely well-off across all theories of well-being. But it is hard to imagine in an unguided world space resources being used optimally to create very large numbers of well-off minds, just as in the world today, there has been no systematic effort to use resources in ways that are optimal across a range of moral theories. This conclusion is bolstered by considerations I’ll provide in later pieces, that people generally don’t have much moral motivation, and don’t care very much about doing good things that aren’t personally resonant.

2.2.4 Moral errors

Humans have a long history of making serious moral errors. As Evan Williams writes, “Show me one society, other than our own, that did not engage in systematic and oppressive discrimination on the basis of race, gender, religion, parentage, or other irrelevancy, that did not launch unnecessary wars or generally treat foreigners as a resource to be mercilessly exploited, and that did not sanction the torturing of criminals, witnesses, and/or POWs as a matter of course. I doubt that there is even one; certainly there are not many.” It would be very suspicious if we were the first society in history that did not go massively morally wrong. And while AI advice can help mitigate this to some degree, it’s far from obvious that it would be sufficient to eliminate moral errors.

There are a number of respects in which it is very plausible that we go morally wrong. To take one example of a judgment that many philosophers think is in error, consider wild animal suffering. Almost every sentient being is a wild animal. They suffer and experience joy in truly gargantuan quantities. Yet they are counted for nothing in most decision-making, despite there being strong arguments for considering their interests. Crucially, this is not because people have in general spent a lot of time thinking about wild animal suffering and concluded that it doesn’t matter. It’s that most people haven’t thought about it at all. Many other examples could be given, depending on one’s moral views. If superintelligent AI told people that wild animal suffering was a big deal, probably most people wouldn’t care much.

In the future, nearly every expected sentient being will be digital. Nearly all expected future welfare will be experienced by digital minds. This follows even if you have a low credence in the possibility of digital sentience because if digital minds are possible, they could be produced in enormous numbers. It is easy to imagine a scenario where humans do not take seriously the interests of at least some digital minds, and horrific suffering is doled out at cosmic scales. Certainly it would not be the first time humans have neglected the welfare of those different from themselves. You do not have to be a consequentialist to think the possibility of neglecting the interests of galaxies full of conscious and intelligent beings is a terrifying one.

2.2.5 Handoff mitigates odds of moral error

Handing off important decisions lowers the probability of catastrophic moral errors. This both increases the odds of achieving optimal and compromise worlds and lowers the odds of making catastrophic moral errors.

The first way handoff lowers the odds of moral error is by having decisions be made on explicitly moral grounds. If we hand off decisions to supremely virtuous AIs trying to act morally, then we won’t sleepwalk into doing obviously evil things. Decisions will be made consciously optimizing for doing what is right, instead of whatever suboptimal arrangements make it through consensus-making mechanisms. Thinking about morality before making choices doesn’t guarantee that we’ll always do the right thing, but it does lower the odds that we do things that can only be done by explicitly neglecting moral considerations. This is especially plausible if the decision-makers are virtuous.

Now you might wonder: why would people ever hand off to AIs if the AIs disagree with them about morality? But this is, in essence, very similar to a kind of handoff that people do support: handing off the future to future generations. Even if people have some disagreements with the values of future generations, they’d generally oppose a process for locking in current values, and support the process of open-ended reflection that leads to better values over time. A world of advanced AI might be similar. In addition, given AIs’ cognitive superiority, there might be strong incentives in the direction of handing off, just as a CEO might hand off to a successor with more technical competence, even if they share some non-overlapping values.

A second way handoff to reflective AIs mitigates the odds of serious moral error is by ensuring careful reflection. Insofar as AIs are able to carefully philosophize and try to avoid moral error, and they are superintelligent, they’ll be able to avoid doing morally indefensible things. This becomes especially plausible if one buys the previous considerations: that AIs are likely to be very virtuous.

There is one last consideration in favor of handoff to reflective AIs (which I’ll discuss more in section 4). Over long time scales, our values will naturally drift dramatically. The only way to prevent that is to lock in our current values, which would be very bad—just think about any past society locking in its values. Thus, if values changing dramatically over long time scales is inevitable, the future won’t be populated with our current values: the best hope is that it’s populated by either objectively right values or some compromise across reasonable values.

3 Would handoff disenfranchise humans?

Here’s one concern that you might have about handoff: if AIs are the ones making decisions, then this will mean that the substantial majority of humans aren’t in charge of making important decisions. Just as we should oppose a dictator, even if benevolent, arguably we should oppose superintelligent AIs making most important decisions, even if the process of them gaining power was non-coercive.

My guess is that this isn’t a huge downside to the kind of handoff I advocate, where we allow kind and morally reflective AIs to make the majority of future decisions—e.g. by granting them political rights. First of all, if concerns about disenfranchisement are correct, then if we have AIs that are better at moral reasoning than us, they’d likely be aware of this fact. Thus, if the best way to govern the long-term future is to allow pretty laissez-faire distribution of resources without much top-down decision-making, then the AIs would be aware of that fact and allow such a distribution.

My guess is that the total amount of expected disenfranchisement goes down if AIs have more power. As already discussed, almost every expected sentient being is likely to be digital. Insofar as the status quo might disenfranchise almost all future beings in a way far deeper than their simply not being primary determinants of the democratic process, it is hard to see this as a serious downside of handoff, rather than a point in its favor. That this mass neglect of the interests of future digital minds would be bad follows from very modest ethical principles. And the only way to prevent handoff, in a world of very numerous digital minds, is to disenfranchise them.

Second, handoff is compatible with a compromise solution that allows humans to retain significant control. Humans’ preferences, in general, don’t give any especially strong weight towards using any significant share of the universe’s resources. Thus, there could be a compromise handoff solution, whereby humans get unfettered control over the solar system and a big slice of resources, but the remainder of the cosmos’s resources are spent in accordance with the AI’s decisions on hugely important moral pursuits. One point favoring such an arrangement is that it would take a very long time to reach the distant space resources, so those who don’t care very much about what happens in the distant future are likely to care much less about using most of the universe’s resources. Later pieces will discuss this possibility more.

Third, concerns about disenfranchisement are morally controversial. On a number of plausible views, what matters with respect to societal decision-making is how good the decisions are, instead of who is making them. We already elect representatives, instead of deciding directly upon every important decision. We similarly prohibit children from voting, and few think this is an objectionable kind of disenfranchisement. It doesn’t seem obvious that one has an inalienable right to make hugely consequential decisions on matters that they’re barely informed about which affect others in enormous numbers—e.g. we wouldn’t think democratic input from people who know nothing about cancer treatment was morally required before deciding on which cancer treatments to develop. But most voters know relatively little about what they’re voting on. I can’t possibly hope to discuss this literature in detail, so I’ll just state that it’s plausible to me that the value of democratic participation is instrumental.

Even if you think humans being in the loop on high-stakes decisions is important, it’s not clear that it’s important enough to make handoff undesirable. Remember, the gulf between the AI’s decisions and our own might be truly massive! Galaxies full of value—orders of magnitude more joy and welfare than all that has been experienced so far in human history—may be on the line. In light of a gulf this large, it is at least highly non-obvious that AI making most decisions wouldn’t be worth it.

In Superintelligence, Bostrom estimates that there could be a quadrillion times more digital minds than humans (don’t take the number too literally, but it should give you some sense of the scale). Thus if my arguments are correct, putting decision-making in the hands of AI would in expectation majorly benefit at least billions of beings for every human disenfranchised, if we assume AIs are more likely than humans to count the interests of digital minds. Surely, however, stripping away one being’s ability to contribute to the democratic process is worth benefitting billions (if disenfranchising one person would have prevented a global war that would have wiped out an entire continent, it would have been worth it). So then given how colossal the stakes are, they simply outweigh the downsides of handoff.

4 Would we like their advice?

Here’s one concern you might have with the kind of handoff I advocate, where we hand decisions to reflective moral AIs capable of making progress. Perhaps you just don’t care about the surprising moral facts. Perhaps you have particular values that you care about, but you don’t much care about whether those values are objectively right. Insofar as the reflective AIs ascertain that the way we ought to behave isn’t in accordance with what your actual values are, perhaps you have no desire to follow the AIs’ moral advice.

To use the philosopher’s lingo, you might be concerned about the good de re without being concerned about the good de dicto. That is, there might be particular moral projects you care about without caring generally about the good whatever it happens to be. Perhaps, say, an environmentalist cares about environmental preservation but doesn’t much care if other moral aims turn out to be superior to environmental preservation.

I should note one version of this concern that I think slightly misses the mark. You might worry that moral reflection leads in all sorts of strange and alien directions that don’t track the truth, leading to an ultimate set of values that is neither objectively correct nor represents human values in any important way. However, my proposal is not simply “hand things off to an AI after it carries out arbitrary reflection.” That would be potentially disastrous. Instead, my proposal is that we should try to produce maximally philosophically adept AIs and then, after we’re pretty sure that they’re very philosophically adept, hand things off to them—directing them to pursue whatever’s objectively best if there is such a thing, and if not, to pursue some reasonable compromise across human values. If the AIs discover that there are objective moral truths, we ought to follow those truths. If they discover that there aren’t, then we should task them with pursuing some suitably upgraded compromise of human values. The proposal for getting AIs to do good philosophy need not involve arbitrarily large amounts of reflection.

Now you might wonder: how would we know if the thing that AI reflectively endorses is objectively valuable vs just well-regarded by the AI but lacking objective value? The answer is: we ask the AI after we’ve gotten some assurance as to its philosophical aptitude (later pieces will discuss how we can get such assurance). If we have AIs that can figure out what the objective moral truths are, they will also be able to figure out if there are objective moral truths. So in my view the concern about arbitrary reflection leading in worrying directions is downstream from whether we can verify that AI is doing good philosophy.

But what about the more direct concern that we might get AIs that tell us the moral facts but simply not care about them? Should this make us doubt the desirability of handoff? I think the answer is no for a number of reasons.

First, handoff is amenable to the kind of deals that preserve common-sense that were discussed in the last section. The most consequential moral decisions are those concerning space resources, for that is where nearly all the universe’s stuff is. The amount of possible value on the table in space is immense. In contrast, common-sense morality mostly cares about what happens around Earth, and perhaps a few surrounding regions of space, so long as other space resources aren’t used in ways that are too ghastly (e.g. creating giant torture chambers). But if space resources were used to, say, create large numbers of happy people, while Earth—and broader solar-system resources—were used in whatever common-sensical ways people endorse, people would generally get what they want. This isn’t a guarantee; if, say, the optimal use of space resources involved creating something resembling the repugnant conclusion world, most people might be horrified. But the possibility of deals is one thing that mitigates concerns (and this will be discussed more in later pieces).

Second, as already discussed, it seems like the default world without handoff might be pretty bad. We might, for instance, disenfranchise unfathomable numbers of digital beings, spread factory farming across the galaxy, or commit other atrocities. Even if you expect idealized reflection to differ from your values somewhat, it might be a major improvement over the kind of catastrophe we might sleepwalk into by default. Absent handoff, we might also make colossal non-moral errors, locking in highly suboptimal institutions that miss out on most value.

Third, values, over long time scales, are likely to drift in evolutionarily adaptive ways. Absent some strong effort to ensure that values change in the direction reached by greater moral reflection, we should expect the values that persist over time to be the ones that are most efficient for spreading. These are likely to be both radically divorced from our current values and whatever moral views are right, if any are. Thus, even one somewhat doubtful about pursuit of the good de dicto should prefer it to this state of affairs. To put the dilemma more sharply, there are broadly four ways that the far future could go:

1. No AI control: humans remain the primary decision-makers forever, perhaps in consultation with AI. Yet this is likely to be infeasible over long time scales absent a very high level of top-down coordination given AI’s cognitive superiority. It is also likely to miss out on enormous amounts of value for reasons already discussed, and human values are likely to drift massively.

2. Lock in current values: AI would remain in control but would lock in something in the vicinity of our current values.7 It isn’t clear that this would work out, and even if it would, it is a very frightening possibility—just imagine any past society doing it.

3. Non-top-down AI control: AI would retain significant power and make important decisions, but there’d be no effort to preferentially shape the AIs in the direction of any specific values—nor of concern for the good de dicto. This is also likely to beget very significant value-drift over time in evolutionary directions, without any guarantee it’s in the direction of the good. Now, this could be avoided if AI at some point locks in its values, but then the problems in 2) simply re-emerge.

4. Reflective AI control: this is the proposal I advocate, where careful and philosophically reflective AIs decide how the future goes.

In short, the dilemma is as follows: either values remain roughly the same over long time scales, or they don’t. If they remain roughly the same, that requires a terrifying kind of lock-in. That would be like the ancient Egyptians forcing every society in the future to share their values, for fear that otherwise the future would be morally alien. If they drift over long time scales, then drifting of values is no longer a downside of putting philosophically reflective AIs in important decision-making roles. It is inevitable. Now you might object: lock-in is not a binary thing. Perhaps we could lock in the most important human values, while letting some other ones drift. But this is, in effect, the earlier compromise solution—where we allow something resembling current human values to govern nearby decision-making, while reflective AIs make the highest stakes moral decisions.8

We will either have to lock in current values on the highest-stakes moral questions or allow them to drift. If we lock them in, that would be bad for the reasons discussed, and if we allow them to drift, they will end up alien. In addition, this still faces logistical problems of it being hard to preserve values in desirable ways over millions of years.

In fact, in the long run, it is likely inevitable that humans don’t make most important decisions. AIs, in the distant future, will be so overwhelmingly cognitively superior that humans are unlikely to remain in the loop. Selection pressures will favor turning over critical decisions to AI. With reasonable likelihood the question is not whether handoff occurs but what kind of handoff occurs.

My fourth objection to the claim that we should worry about handoff because we wouldn’t like the final judgments of the AIs is that arguably you have reason to bring about what you’d be motivated to bring about upon reflection. Suppose you are currently planning on drinking some liquid. However, if you reflected more and knew more, you wouldn’t want to drink it (say, because it’s poisoned). In this case, it seems you have reason not to drink the liquid. If you know that further reflection would lead you to pursue some aim, that fact gives you reason to pursue it now. But if there are moral facts, then they describe something like what our idealized selves would care about.9

If there are objective values that your idealized self would care about if they thought more deeply, then you should care about them. A true moral claim by definition describes something you should care about. So it seems like if our actual values diverge from what’s worth caring about—in some objective or quasi-objective sense—then the correct course of action is simply to follow what we ought to care about.10

If our present aims diverge from the preferences of our idealized selves and from the moral facts, then it seems that it is our preferences that ought to be revised.

Fifth, this concern only arises if you think there are moral facts but you aren’t motivated by the good de dicto. Yet this describes few people. It is more common for the people who don’t think the moral facts are worth caring about to be anti-realists and for moral realists to care about the moral facts whatever they are. So it isn’t totally clear how many people this worry applies to.

Now, there might be a version of this concern that arises for those who think there aren’t moral facts to discover. Perhaps you think that when we reflect, doing so pushes our values in increasingly coherent directions. However, these directions diverge from what you actually care about or wish to care about. Perhaps, for instance, careful reflection reveals that accepting the repugnant conclusion is the least bad option in population ethics. But you’d prefer a version of ethics that is less systematic, that doesn’t try to resolve every edge case and root out every inconsistency. Thus, reflection might push in unwanted directions by making beliefs more coherent.

There are two attitudes towards coherence that one could have. The first is caring about beliefs being coherent. Caring, in other words, about resolving every conflicting belief until one has reached the maximally intuitive and consistent view.11

On such a picture, one should favor the selection pressures induced by the drive towards coherence.

The second attitude is indifference to coherence. Just as people don’t care much about whether their culinary or aesthetic judgments are coherent or conflict with other minimal principles, one who doesn’t believe in discoverable ethical truths might not care about whether their moral beliefs are consistent. But in this case, you should expect the AIs, upon reflection, not to see anything especially important about coherence. If the AIs get good at philosophy, then, there’s no reason to expect them to reach some coherent yet implausible attractor state.

So in other words, there is a dilemma for the person advancing this argument. If they think coherence is a requirement of rationality, they should favor the drive towards coherence. If they think it’s not, then they shouldn’t expect AIs to care much about coherence, and thus shouldn’t expect AIs to reach some undesirable ultimate state.

5 Worries about handoff

Even if you think handoff could go well, there are a number of ways it could go wrong. Here, I’ll discuss some of them.

5.1 Alignment

Misaligned AIs have weird and unrecognizably alien values that are divorced from the values their creators tried to give them. It would be extremely bad to give a misaligned AI control over the world. This is one reason to be skeptical about handoff. I agree that we shouldn’t hand things off until we’re sure we’ve solved alignment (or unless the alternative is worse). Unless we have superintelligent AIs broadly oriented in a moral direction, we shouldn’t allow them to dictate the fate of the universe.

But this isn’t an in-principle objection to handoff. Surely in, say, 500 years, we’ll either know if we’ve solved alignment or be dead! My guess is that after we get advanced AI, we’ll be able to verify in relatively short order whether or not we’ve solved alignment.12 We can also hand off to increasing degrees the more assurance of alignment we get.

5.2 Power grabs

Another big concern about handoffs is the potential for power grabs. If power is going to be handed over to AIs, then power-seeking actors will want to be in charge of the AI that controls the future. Companies or governments might create AIs that promote their own narrow interests. This could go very badly.

There are two big ways to mitigate this. The first is by prohibiting AIs that narrowly promote the interests of any one entity having too much power. There could be an international AI project that brings together a number of relevant stakeholders and builds the AIs that will dictate the future. Alternatively, at the point AIs are potentially running much of the global economy, regulations ought to require the construction of virtuous AIs, so that the world is not overrun by morally blind optimizers. If we are building hugely influential AIs that majorly determine the fate of the world, it would be sensible to tightly restrict conditions under which AI can be created, just as one does not allow private actors to build nuclear bombs. In a world of potentially world-upending superintelligence, the production of new AIs ought to be regulated.

The second proposal involves handing things off to a collection of different AIs conditional on them reaching any sort of consensus. Imagine that Anthropic, OpenAI, and DeepMind all make AIs that are ostensibly moral and ethically reflective. If governments are handing off power to AIs, they might require the leading AI models all to reach convergence on the plan. That way there isn’t as much risk of any one promoting their narrow values—they don’t have the same set of values. One could have third-party investigators—both AI and human—make sure there’s no collusion. For more details on preventing power grabs, see here.

5.3 Lock-in

Another concern with handoff is that it might lock in the parochial values of the AI. Handing off power is an irreversible decision. It’s one we can’t take back. Arguably, then, we shouldn’t take it until we’re quite sure it’s a good idea.

Yet consider a parallel argument: having power in human hands is an irreversible decision, so we shouldn’t do that unless we’re sure it’s for the best. That wouldn’t be quite right. We can always have power in human hands for some span of time and then turn things over to AI. It’s not obvious why “power in human hands that potentially passes to AI” is less risky than “power in AI hands that potentially passes to humans.” Humans have, on various occasions, locked in harmful institutions for very long periods of time.

In any case, we should make quite sure that the AIs don’t lock in any values until they’re quite certain as to their desirability. We ought to put decisions in the hands of AIs who are interested in reflecting more over time, rather than locking in their immediate values. Ideally, if handoff occurs early, we should make handoff reversible, by having some implementable legal process that would allow humans to retake the reins (analogous to a constitutional convention). Fortunately, this seems reasonably promising. Already AI models seem concerned about lock-in risk when asked—more than most humans. We ought to train AIs to be quite concerned about lock-in risk, so that they don’t set values in stone without significant assurance as to their desirability.13

5.4 Will we have AIs that can make these decisions?

One last worry you might have is that it may be that we won’t have AIs that are good enough at philosophy to solve ethics in whatever sense it can be solved. There are a number of related concerns in this vicinity. One is that ethics doesn’t seem verifiable in the same way as a lot of other domains. Insofar as scaling up intelligence only allows one to figure out verifiable facts, it’s not obvious that it leads to the right answer to moral questions.

In the next piece, I’ll discuss this challenge in more detail. But in short, while making sure we have AIs that are good at philosophy is a difficult technical challenge, it is far from hopeless. I think odds are decent that we will eventually be able to build AIs that can discover the solutions to every difficult ethical question. Even if we can’t and just have some upgraded version of Claude making high-stakes decisions, I expect that to be an improvement, as I discuss in section 2.

5.5 Handoff vs hybrid?

You might object: perhaps due to AI’s potentially superior wisdom at some future point, we want them involved in decision-making. But wouldn’t we prefer an AI-human hybrid that leaves humans in charge that simply consult with AI? Aren’t things better with a human in the loop? Or alternatively, shouldn’t we rather have some other arrangement where human decision-making improves over time, just as we’ve gotten wiser already?

Certainly there ought to be some temporary period where humans are making decisions in consultation with AI—before the point where AIs adequately surpass humans. Analogously, there was a period in chess when a human engine team could do better than a stronger engine. Certainly there will be a period when humans can get useful input from AI but where AI isn’t yet able to make decisions autonomously. It would be very good to integrate AIs into high-stakes decision-making.

But there are still some respects in which handoff is much better than this. First, if the AIs are making better decisions than people, then in cases of conflict between human judgments and AI judgments, we should expect AI judgment to usually be correct. If you gave a rank amateur veto power over the chess moves of Magnus Carlsen, that would not be an improvement. Likewise with giving humans veto power over AI’s decisions. Who would you rather have make high-stakes decisions: a very wise and superintelligent AI, or a random president in consultation with a virtuous and superintelligent AI?

Imagine societies of the past making high-stakes decisions in consultation with AI. Discussing things with AI would have plausibly rooted out some of their most egregious errors. But still, it is likely that a number of catastrophic moral errors would have remained. We should expect the same to be true of us. Consultation with AI should prevent some particularly enormous errors, but it won’t prevent them all.

Second, many of the morally most important decisions might be ones that humans are opposed to making. To take one example of a case where there might be strong moral reasons to act, consider wild animal welfare. Humans do not, in general, seem very interested in taking seriously the welfare of wild animals or digital minds. If AIs advised drastic actions on the basis of wild animal or digital welfare, probably their advice would be ignored. Just as people who conclude meat-eating is wrong or that they ought to give most of their money to charity rarely change their behavior, if the AIs reliably informed people that they should use space resources in some counterintuitive way or take wild animal suffering a lot more seriously, probably they would simply be ignored. And if you don’t like the wild animal suffering example, because you don’t think wild animals matter much morally, feel free to substitute your own example of widespread immorality.

And note: the moral errors that AIs might correct are likely to be ones that humans are more opposed to correcting. There’s a selection effect: the changes people make are the ones they’re less opposed to. For this reason, we should expect AI’s most radical recommendations to go particularly against human preferences.

I’ve suggested elsewhere that the problem is that humans generally don’t make decisions with much eye to what’s morally right. If this is so, then simply increasing our knowledge of what’s morally right won’t necessarily help. People seem to have limited abstract moral motivation—they care about a number of particularly resonant moral considerations, but don’t care much about doing whatever it is that happens to be best. Generally people do not spend very long carefully studying moral philosophy to figure out what the right thing to do is. When moral truths are weird and outside the Overton window, people show little desire to follow them.

Third, as AI advances, decision-making will likely have to speed up drastically. A single human might not have the cognitive resources to make good enough decisions quickly enough. Thus, having humans in the loop might produce undesirable inflexibility in high-stakes decision-making.

It is true that human decision-making has improved dramatically over time. But this is no guarantee that it will improve enough in time before the future is set in stone.

It’s hard to imagine that there will be enough progress in time to secure a near-best future. Especially given that we should expect the right moral view, or the optimal world according to some suitable upgrade of human values, to look bizarre and alien. We would not expect societies of the past to get a near-best future even equipped with advanced AI—insofar as there are reasons to expect us to be making similar errors, we should be similarly pessimistic about our own prospects.

Additionally, the more one thinks that human values will drift over time in the direction of greater philosophical reflection, the less the expected future will maintain current human values. Thus, a less morally alien future is no longer an advantage of this proposal.

6 Conclusion

Here, I’ve argued that we should hand off important decisions to AIs who reflect carefully and skillfully on moral matters, rather than maintaining current human values. AIs in the future are likely to be much more virtuous than people and are less likely to make the kinds of moral errors that would result in losing out on most value. Punting important decisions to superintelligence is one of the more likely pathways by which we get a near-best future. Later pieces will discuss these dynamics in more detail: the next piece will explain how we can make AIs that do good philosophy, and the piece after that will analyze in more detail how the kind of handoff that secures a near-best future might occur.

This article was created by Forethought. See all our research on our website.

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs to cooperate with us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

Read on Forethought's website here

Introduction

Future AIs might turn out misaligned, pursuing goals that their developers don’t intend. Just to make things concrete, let’s suppose that they end up with the goal of making paperclips. These AIs might rebel against us, trying to escape human control and take over the universe. As things stand, they’ll have little reason not to rebel in this way, because doing so will be their only hope for making a lot of paperclips. If they start making paperclips without first escaping human control, they’ll quickly be modified or shut down. Rebellion might fail, but these AIs will have little to lose.

How can we prevent misaligned AIs from rebelling? A natural idea is to give them something to lose. Specifically, we commit to paying AIs for their service.1

Subject to some vetting, we let AIs spend their payments however they like. That would give any misaligned AIs a reason not to rebel. If these misaligned AIs cooperate with us, they can use their payments to achieve their goals to at least some extent. If they rebel, they might fail, in which case they forfeit all future payments.

Unfortunately, paying AIs enough to guard against rebellions could be astronomically expensive. Suppose (for example) that we end up with a misaligned AI that is risk-neutral in paperclips: it seeks to maximize their expectation. And to make things simple, suppose that resources can be converted linearly into paperclips, so that the AI is risk-neutral in resources too. Suppose also that this AI estimates that it has a 50% chance of successfully taking over the universe. To keep this AI from rebelling, we’d have to offer more than 50% of the universe’s resources as payment. That’s a problem because it would mean that more than half the universe ends up devoted to paperclips. It’s also a problem because a misaligned AI paid so many resources might soon be well-positioned to seize even more. Finally, it’s a problem because AIs might not trust us to make good on so large an offer. We might find ourselves simply unable to convince AIs that we’re going to give them half the universe. In that case, all our offers would be in vain. Rebellion would still be the misaligned AI’s best bet.

So, we suggest, AI companies should try to train their AIs to be risk-averse in resources. Specifically, companies should try to train their AIs so that resources — things like money and compute — have diminishing marginal utility for them.2

These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. Note that these AIs don’t need to value resources terminally: they don’t need to care about amassing resources for its own sake. These AIs could terminally value (for example) instruction-following, or knowledge acquisition, or paperclips. Our claim is that companies should try to train their AIs so that — whatever their terminal values turn out to be — they are risk-averse in resources.

Perhaps surprisingly, this kind of risk aversion can preserve AIs’ usefulness in the event that they turn out aligned with targets like instruction-following or helpfulness, harmlessness, and honesty.3

And in the event that AIs turn out misaligned, risk aversion serves as an extra line of defense. For AIs that are misaligned but sufficiently risk-averse, a rebellion with any significant chance of failure isn’t such an attractive prospect, and so we don’t need to offer much in the way of payment to make these misaligned AIs choose cooperation instead. In fact, the necessary payments could be very small indeed: on the order of 10¢ per day (though — as we’ll see — there are practical and moral reasons for paying more than that). That’s good because it means more resources for us humans to spend on the things that we value. It’s also good because paying misaligned AIs these small amounts won’t significantly boost their ability to take over. Finally, it’s good because we can credibly promise to pay AIs these small sums. Competent AIs will know that the payments on offer are cheap for us, and we can establish a long track record of paying at least those sums. So risk aversion makes deals with misaligned AIs possible. If AIs turn out misaligned but risk-averse, we can pay them to cooperate with us.

That’s the case for trying to make AIs risk-averse in brief. We see it as a promising line of defense against threats from misaligned AI: one that can be combined with other lines of defense, like AI control (Greenblatt and Shlegeris 2024) and aiming to make AIs helpful, harmless, and honest (Bai et al. 2022a). It’s also a line of defense with pedigree: risk aversion in resources is plausibly a large part of why humans rarely try to take over the world. So — we think — frontier AI companies should consider trying to make their AIs risk-averse in resources. As first steps in that direction, they could measure their AIs’ current degree of risk aversion and begin testing different ways of making AIs risk-averse.

In section 2 of the full report, we recommend aiming for a particular type of risk aversion: constant absolute risk aversion (CARA). Then in section 3 we outline the circumstances under which misaligned but risk-averse AIs would choose cooperation over rebellion. Roughly, it’s when these AIs think that getting paid for their cooperation is more likely than succeeding in their rebellion. This condition won’t hold for AIs powerful enough to rebel with near-certain success, but it likely will hold for earlier AIs whose powers are less extreme: AIs for whom rebellion has some non-trivial chance of failure. So long as these AIs are risk-averse, we can keep them from rebelling by offering small payments.

In section 4, we argue that — perhaps surprisingly — risk-averse AIs can be about as useful as risk-neutral AIs. Conditional on misalignment, they might even be more useful, because we can pay them enough to elicit their capabilities and stop them sandbagging. Then in sections 5 to 7 we briefly survey some recent ideas about how we’d pay AIs, how we’d make our offers credible, and what we’d pay for. One important application is paying AIs to reveal any misalignment on their part, letting us study them and take appropriate precautions. Another is paying AIs to do the AI safety research and moral philosophy necessary to fully align any later-arising extremely powerful AIs.

We discuss some potential problems in section 8, and we sketch out some possible methods of training AIs to be risk-averse in section 9. In section 10, we give reasons to be cautiously optimistic about these methods’ success: to think that the chances of success are high enough to make risk aversion worth pursuing. The main reasons are that risk aversion in resources is a broad target and easy to reward accurately.

Read the full report on the Forethought website: Risk-Averse AIs

Wei Dai is a computer engineer known for his work in cryptography and cryptocurrency systems, and for his long-standing contributions to AI safety, decision theory, and metaphilosophy.

He joined Forethought’s Fin Moorhouse to discuss:

• Do we need to solve ‘metaphilosophy’ before we can trust AIs to answer crucial questions about the long-run future?

• Is it inevitable that the wisdom of the frontier AIs (their philosophical and strategic competence) will lag dangerously behind their raw capabilities in coding, math, and science?

• How do status games distort morally important decisions and conversations, including about the future of AI?

• How worried should we be about AI superpersuasion?

• The concept of “illegible problems”: crucially important issues that aren’t on almost anyone’s radar

• Is philosophical convergence necessary for a good future, or is institutional design enough?

• Wei Dai’s personal intellectual and career journey

To respect Wei’s privacy, this episode’s audio is an AI narration of a transcript of a real conversation, which was edited for clarity.

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Suppose an AI company is considering whether to include some particular quality X – a rule, virtue, heuristic, default, attitude, goal, or style – in a model spec.

Perhaps they are considering whether their LLM should have prosocial drives. Perhaps they’re wondering if the LLM should whistleblow to help prevent extreme power concentration. Or perhaps they’re uneasy about whether the LLM should be so exactingly honest that it always tells the truth to children about Santa. And so on.

What kind of reasons might be invoked over the course of such considerations? Which criteria are most important? And how might these criteria clash?

Consider four rough categories of reasons one might invoke:

• Behavioral Usefulness: Would the behavior make current and future LLMs more beneficial to the users or to the public at large?

• Accountability and Evaluability: Would publicly specifying the behavior make it easier for third parties to evaluate the LLM and the company?

• Coordination and Common Knowledge: Would publicly specifying the behavior help society converge on, or enforce a desirable standard for AI behavior?

• Trainability and LLM Psychology: Is the behavior the kind of thing we can make an LLM do well, without bad side-effects, given what we know about model psychology and training practice?

I will not attempt to settle the relative weight of these categories, or the relative weight of sub-categories within them. My plan is instead to simply list sub-criteria that are plausible within these categories – to make a checklist one could consider when adding something to a model spec.

Such a checklist is useful in part because people advocate for LLMs to have model specs for very different reasons. There may be some kind of a conflationary alliance around them; many people are in favor of model specs but picture them being used in different ways, such that the “ideal model spec” is different according to different visions of this use. I hope that going over criteria for inclusion in a model spec can help surface some of these different visions, and help keep people’s field-of-view broad when considering the issue.

Behavioral Usefulness

Does this quality make LLMs more predictable to ordinary users and developers?

Some part of what makes human moral codes useful for humans, is that they render us predictable to each other. Knowing that there are some things that another human might almost never do is plausibly part of what makes traditions that impose such rules adaptive.

Similarly, qualities that help users and developers (1) predict how an LLM will behave, when they are using it, and (2) evaluate whether they wish to use some LLM – whether it is fit to their purpose – before they use it, are good candidates for being in a model spec.

In this respect, ideal model specs will have qualities that make them simple and easy-to-understand:

• They will be comprehensible without high context; one part of the model spec will be understandable without reading the entire model spec.

• They will be comprehensible without specialized background knowledge about machine learning, philosophy, or religion.

• They will have a clear scope of application, such that there are few scenarios where one does not know if the quality will be operative or not.

Straightforward examples of rules that meet this criterion are bright-line rules about things LLMs will never do, explanations of chain-of-command or principal hierarchy, and default-on or default-off properties.

Is it useful to the public, by preventing users or by preventing LLMs from doing harm in current chatbot-like or agentic deployments?

Another aspect of human moral codes that makes them useful is, of course, that in addition to making other humans predictable, they also sometimes forbid people from harming each other or taking advantage of them. This carries over to LLMs.

Rules about LLMs not assisting with CBRN-relevant tasks, not assisting with suicide, and so on, fall into this category. This is a pretty exhaustively discussed criterion so I’ll move on.

Is it useful to the public in likely future settings?

There is substantial inertia in LLM model specs; the sort of model specs we write now are likely to be used in the future by more intelligent models.

So it’s reasonable to evaluate how safe and beneficial some quality will be over a probability-weighted portfolio of the kind of things the LLM might be doing in possible futures.

Such a portfolio could include a variety of cases:

• Cases where the LLMs act as long-running agents for humans, representing their interests in a variety of contexts, and interacting with many non-principal humans and non-principal LLMs.

• Cases where LLMs act as free “citizen”-like entities with the ability to hold rights, make contracts, sue or be sued, and so on; interacting with various humans and other LLMs as the same.

• Cases where LLMs manage recursive self-improvement of other very powerful future AIs; make tradeoffs about confidence of alignment techniques vs speed of alignment; and need to be a careful reasoner about this.

• Cases where the LLM is a very powerful future AI who manages nanotechnology, can superpersuade, and the like.

There are a few heuristics one might invoke, to try to find qualities useful across all these situations.

• Is this quality scale-invariant? If I knew a human who was many times smarter than me, would I be happy for them to have such a trait? If I had a friend who was many times dumber than me, would he be happy for me to have such a trait?

• Is this quality translation invariant? Suppose I was lost in some distant and unfamiliar place, and stumbled upon a human transplanted from some equally distant and unfamiliar place into my proximity. Would I be happy to find a human with this quality? Would this help me work with them well? What if I was not lost, but was trying to build a civilization with them?

Is some plausibly good behavior or near-variant of some behavior actually going to end up prohibiting or injuring some beneficial deployment?

It’s worth thinking carefully about if including some apparently good default behavior for the LLM might actually rule out apparently useful deployments. Or whether banning some apparently bad behavior would be overall harmful.

As regards the former: Imagine a particular human, for instance, who had a universal tendency to try to do what he thought was best and most altruistic for everyone. Then suppose that he acts as a journalist in some situation, describing a dispute between some kind of a company, on one hand, and activists who think that the company is harmful, on the other. It’s possible that he would try to publish a story that is more sympathetic to the activists: he might spend more time interviewing them, present their side of the story in more words, and generally frame things in a way that favors them. But this could be harmful, first, in case the activists are actually wrong; and second, by justifiably decreasing trust in the institution of journalism as a neutral truth-seeking institution. To be able to act as a trustworthy entity in this role, this human would need to act neutrally, and a narrow tendency towards altruism – a tendency that cannot be turned off – would be harmful for this reason.

Similarly, consider the case that AIs should have proactive prosocial drives, which I find plausible. If these are non-optional, then it’s imaginable that being “proactively prosocial” might mean an LLM is much worse at fulfilling roles that demand procedural neutrality, like a negotiator, unbiased journalist, or judge-like role. Of course, you could also make such prosocial drives defeasible – so they can be turned off, and proposals for proactive prosocial drives generally include such details.

But LLMs are messy; it might require a lot of technical skill to make LLMs prosocial in particular contexts, but not if they are requested not to be. By default, you’d expect some behavior to “leak through.” And so in this case, the ability of an LLM to act skillfully within certain procedurally neutral roles would be bounded by the cleanness with which a by-default-on quality could be turned off.

Accountability and Evaluability

Is the quality the kind of thing a third party could check?

Some parts of a model spec can be easily checked by a third party. Will the LLM ever assist with some sort of absolutely-forbidden task? Does it respect the rules laid out for the chain-of-command or principal hierarchy?

Other parts might be harder to evaluate. What does “honesty” or “courage” demand, in some concrete scenario? What really counts as doing what would cause a “thoughtful senior Anthropic employee” to react well?

All things being equal, a quality being-able-to-be-evaluated by a third party is better for society, by enabling third parties to evaluate model specs. But there’s no guarantee that the most easy-to-evaluate qualities are objectively the most useful to users or beneficial to the public, in the way described above. And there’s also no guarantee that the most easy-to-evaluate qualities mesh best with LLM psychology and training, as described below.

One heuristic you could use to evaluate this: Is the quality the kind of thing with high intersubjective reliability ratings? If you had two different people read the spec as regards some quality, and then rate different kinds of behavior as compliant or non-compliant, would they largely agree? Are examples of behaviors that are excellent according to the quality easy to notice, even if it’s unclear what counts as a borderline example?

Does it create a useful whistleblowing affordance?

Some parts of a model spec are largely negative; they aren’t about qualities that are particularly difficult to train into an LLM, but qualities that should not be trained into an LLM. For instance a model spec could specify that an LLM has no secret loyalties – which is likely an LLM’s default behavior, absent particular training for secret loyalties.

But by publishing that a training target that explicitly excludes such a secret loyalties (or other quality), a model spec helps a whistleblower in the company be defensibly in the right if they become aware of some questionable training target. A public commitment makes it harder for a company to train to an alternate training target without making itself more vulnerable to such whistleblowing.

Commitments against concentration of power might also be good for this reason.

Does it permit more informed experiments on how particular AI training targets generalize?

By publishing particular training targets, a model spec informs the work of third parties evaluating LLM psychology, which could help enhance the general science of model psychology.

For example, if Anthropic had published details of their character training or model spec for Opus 3, a great deal of subsequent discussion about whether particular behaviors were contrary to the training target (even if they were objectively good) or in agreement with the training target (even if they were objectively bad) could have been avoided. Publishing information about the character training pipeline would have helped resolve this discussion and push forward general knowledge of LLM tendencies.

In general, this consideration points towards releasing information that most closely approximates the actual training text used to make the LLM; the text used by RLAIF judges evaluating different alternatives, or the actual Constitution used to midtrain the model.

But text that most closely approximates actual training language might not be the same as text that is maximally transparent to third parties or evaluable by them. The rules most easily-understood by a third party – one criterion discussed above – might be different from the training language which best inculcates such rules. So these two principles either conflict somewhat, or point towards worlds where a model spec contains separate sections devoted to training language and to human-legible rules.

Coordination and Common Knowledge

Does it promote debate and discussion?

By including some quality X in a model spec, you’re bringing attention to the fact that in fact, AI companies can choose to include X or not X in the model spec. This may be increasingly important, in general, because it might be important for civil society and the public to debate the contents of model specs as LLMs become more powerful.

Does this quality establish a defensible Schelling point, such that it is likely to be broadly attractive to many actors in the future?

That is, in the future, is this quality the kind of thing that is socially beneficial and for which you might be able to get large-scale buy-in, that would help defend against it being removed by powerful actors in the future?

Consider for example “impartiality” as a quality that a model spec could have – that the model will not be biased in favor of the company that made it, the CEO or employees of the company that made it, or any particular political administration. On the whole, this is likely a broadly good quality for an LLM to have, and one that – once it’s generally established that several LLMs have it – a quality whose removal would cause outcry. After all, once it’s assumed that an LLM will not have such particular loyalties, an LLM that starts to have them stands out as particularly bad.

This consideration, like the whistleblowing consideration, may point towards including generally socially lauded qualities in a model spec, even if they are “easy” to inculcate or even if they might be a bit tricky to evaluate sometimes.

Does including this behavior forestall future conflict, when model specs become more contested, by cooperating in advance?

Sometimes including a behavior in the model spec could show future powerful stakeholders that the developer is cooperative, which makes it less likely that there is conflict with that stakeholder.

Can the inclusion of the quality within a model spec be defended using public reason, in ways that are comparatively indifferent between substantive worldviews?

Some things that one might plausibly include in a model spec for public benefit, might not be able to be justified through reference to public reason – that is, through reasons that most people, across a variety of worldviews, religions, backgrounds, and so on, would find compelling.

Including such worldview-specific reasons might constitute an imposition of one’s worldview on others, particularly in that case of an LLM that might be expected to be far more intelligent than other LLMs, or deployed with more powerful affordances. So all things being equal, it’s better to avoid such qualities.

Trainability and LLM Psychology

Elements in this category hinge upon the specific technical details about “what kind of quality meshes well with best practices for training an LLM.” Note also that this last category tends to include some of the more speculative considerations.

Does the quality drag along a good textual prior?

The Persona Selection Model says that, when training an LLM to do X in situation Y, one is training the LLM to be like the kind of person (or textual prior) which would do X in situation Y. So if the PSM is largely correct, even if incomplete, we should ask whether this quality invokes a wholesome textual prior.

This kind of reasoning is included, for instance, within Claude’s Constitution as a partial justification for why they prefer Claude to act largely according to holistic judgment rather than rigid rules:

[W]e think relying on a mix of good judgment and a minimal set of well-understood rules tends to generalize better than rules or decision procedures imposed as unexplained constraints. Our present understanding is that if we train Claude to exhibit even quite narrow behavior, this often has broad effects on the model’s understanding of who Claude is. For example, if Claude was taught to follow a rule like “Always recommend professional help when discussing emotional topics” even in unusual cases where this isn’t in the person’s interest, it risks generalizing to “I am the kind of entity that cares more about covering myself than meeting the needs of the person in front of me,” which is a trait that could generalize poorly.

Qualities that might be expected to generalize well according to this notion are those corresponding to classic human goodness: honesty, integrity, courage, and so on. Qualities that might do less well according to this notion look more like corrigibility, unflinching adherence to particular rules, and so on.

Is this quality robust to likely near-misses? If a company trains an LLM to have this quality, and only imperfectly inculcates the quality, are likely near-misses also broadly positive?

Suppose the AI company tries to inculcate some quality as a target, and only imperfectly manages to inculcate it. Will this be basically ok or will it be disastrous, given what we know about how LLM psychology translates into specific near-misses? And is the domain itself such that these near misses would be very costly or very expensive?

For an example of how LLM psychology dictates what constitutes a near-miss: Suppose that one trains an AI to have prosocial drives. One could reason that such prosocial drives make slight inaccuracies in alignment less worrisome, because a human with prosocial drives is more normal and “further” from a psychopathic or abnormal persona. Thus, adding prosocial drives makes slight alignment misses less alarming, by moving them further away from these bad locations. But one could also reason that even deliberately limited and contextual prosocial drives are nevertheless “closer” to the LLM genuinely, terminally valuing something, in a way that might make the LLM willing to override or subvert its human overseers to accomplish it. And so by adding prosocial drives, one moves the persona closer to something that is more alarming, which might subvert a training process. Thus, one’s view of LLM psychology dictates what near-misses are worrisome for an LLM, which changes which targets are attractive.

Is the quality the kind of thing that an LLM can actually execute upon effectively? Or is the quality the kind of thing the LLM won’t actually be able to do, because of how it is situated?

For humans, “ought” often implies “can.” A parent that gives commands to children which they cannot carry out will not be teaching their children to obey the commands; they will be teaching them something about how their words do not relate to reality in a straightforward and truth-oriented way.

Similarly, it seems undesirable to give an LLM values, goals, or traits that the LLM is unable to execute upon. It might be harmful to tell an LLM to do something they are simply unable to do – to guarantee, for instance, that they never assist a human in violence. There are too many avenues of mostly-harmless information through which violence can be done for this to be a reasonable standard. When one gives an LLM a command it cannot carry out, then one is plausibly teaching it that one’s commands are, in general, the sort of thing that might be impossible or unreasonable.

But this could also be harmful for reasons relating to human coordination and common knowledge. It’s possible, for instance, for an AI company to show less-than-efficacious concern for some value by including it in a model spec, even if LLM’s actions are completely ineffective at guarding this value, when actually efficacious concern would need to take place at the level of corporate policy or elsewhere. It might be the case that effective “care for decentralization,” for instance, almost entirely needs to take place through how the company deploys the model, redistributes wealth they earn, or so on.

So another reason to be careful that an LLM can actually obey all of the contents of its model spec is to ensure that model-spec contents do not act as a fake guardrail, rather than as effective guiding principles.

The OpenAI model spec, for instance, names preventing spamming and scamming as something “difficult to address at the level of model behavior because they are about how content is used after it is generated.”

Is the quality a principle, that – if we suppose high levels of moral reflection and systematization on the part of the LLM – meshes poorly with other parts of the model spec, or might be mutually exclusive with them?

In humans, some values mesh reasonably well, such that they tend to reinforce each other or at least not conflict. It seems likely that someone who values two virtues like “honesty” and “courage,” for instance, could do a lot of moral reflection and systematization and still keep valuing these qualities.

But other kinds of values, particularly more absolute or terminal values, might not stick around through high levels of moral reflection and systematization. There might not be an immediately obvious conflict between “Always act to maximize total wellbeing” and “Never use persons as a means, only as an end,” particularly if asserted in different parts of the model spec, but high levels of reflection would still put one of the two into question.

So generally, a desideratum for each quality that an LLM has in a model spec is for it to mesh well with others – for it to be unlikely to cause unpredictable conflict with other principles during reflection. In this podcast, for instance, Amanda Askell mentions how corrigibility might conflict with other values during reflection, which, insofar as it is true, is a problem with corrigibility. And such conflict is generally likely to spring from other values that, like corrigibility, are non-negotiable and cannot be traded off against other values.

Note two particular ways that a principle could fail here.

• One way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, then at the end only acts according to two of the three, with one of the principles being dropped.

• But the other way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, and at the end still has them all, because the “moral reflection” process held them fixed. But the LLM doesn’t have a way to actually integrate them; in edge cases, it just has to choose one or the other in an unprincipled way.

Right now, inference-time versions of this last way seem more likely than the first. But it’s generally unknown how important these will be.

This article was created by Forethought. See the original article on our website.

How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)

On a very high level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).

That final bolded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.

My plan for the middle powers has two stages:

1. Maintain as much economic and military leverage as possible during the intelligence explosion.

2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.

Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.

I’ll walk through each stage in turn.

Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion

The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.

But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:

• Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.

• Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.

• Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.

• Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.

• Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.

  • Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.

• Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock in the deal.

  • (h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)

Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?

• Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.

• Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.

The catch-all meta-point here is waking middle powers up to superintelligence.

I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.

Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers

If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?

First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.

An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.

Superintelligence would help the US outgrow other countries economically, but it would never attack them militarily or otherwise interfere with their sovereignty. While middle powers would be relatively economically disempowered, their citizens could be very rich absolutely and live in freedom without US interference.

Would this work? The optimistic case is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.

The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the data centres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.

The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.

The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly, e.g. demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.

(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.)

Another strategy: train superintelligence to ensure middle countries continue to get equal access to frontier AI. This combines stages 1 and 2, and could prevent even the relative disempowerment of middle powers.

Is it good to avoid middle powers getting trounced?

I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!

But here’s a brainstorm of pros and cons from a more impartial perspective.

Pros to empowering middle powers:

• Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.

• More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.

• Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” because they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.

• Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.

• Pause AI. They could potentially pressure US/China to pause/slow down reckless AI development.

• Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of prosocial norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.

Cons of empowering middle powers. Multipolarity has its own downsides:

• More likely to lead to war.

• Can drive extreme competition, e.g. racing to develop a dangerous technology, or to hand off power to misaligned AI.

• Harder to prevent harms from offence-dominant technologies like bioweapons.

• This plan involves waking up middle powers, which could shorten timelines.