Of past algorithmic improvements which decrease LLM pretraining loss, many have been “scale-dependent” – that is, improvements which decrease LLM pretraining loss by more, relative to prior algorithms, at larger quantities of compute.

The influence of such scale-dependent algorithms is (1) moderate evidence that attempts to limit algorithmic progress in absence of compute limitations will be ineffective, (2) weak evidence that our inferences about the future scale of algorithmic progress, based on the past, are invalid, and (3) part of a plausible argument either for or against a software intelligence explosion, depending on other details of one’s model.

I’ll proceed by discussing the following:

1. How scale-dependent algorithms for pretraining loss probably account for the majority of algorithmic improvements in this domain

2. How scale-dependent algorithms for end-to-end task performance might account for the majority of algorithmic improvements in this domain

3. How scale-dependent algorithms in the past should influence our model of algorithmic improvements in the future

4. Possible policy implications

Before starting, a note on method – when discussing questions regarding AI’s past and future progress, there’s a natural continuum from “the intractable, high-impact, abstract question, which we terminally care about” to “the tractable, dubious-impact, concrete question, which we instrumentally care about.”

Consider questions about scale-dependent algorithmic improvements, sliding from the former to the latter along such a continuum.

• We are most terminally interested in questions about the future: “Will future improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? What can we know about how these two will be causally intertwined? Will future improvements enable a software-only intelligence explosion – that is, in a world with total compute held constant, could algorithmic progress lead to an intelligence takeoff in a comparatively short period of time?”

• The data we can gather most relevant to this question is about the past: “Did past improvements to AI task performance depend more on compute scale-up or on algorithmic improvements? Would past AI algorithmic improvement have counterfactually looked slower, without the datacenter buildout?”

• And finally we have the narrower sub-question for which we have the most past data, which contributes to answering the broader historical question: “Did past improvements to LLM pretraining loss depend more on compute scale-up or on algorithmic improvements?”

I’m going to start by discussing the evidence for this last question, before moving to the increasingly impactful and increasingly difficult earlier questions.

Scale-Dependence of Algorithmic Improvements for Pretraining Loss

Are the algorithmic improvements that have most decreased pretraining loss for LLMs scale-dependent or scale-independent? What would either of these mean?

The compute-equivalent-gain (CEG) for some algorithm A’, relative to another algorithm A, is the ratio of the FLOPs needed to train an A-using model to a given level of performance, over the FLOPs needed for an A’-using model to reach that same level. If it takes 3x as much compute to train an LLM with algorithm A to reach the same pretraining loss as another LLM trained with algorithm A’, then A’ has a 3x compute-equivalent gain on pretraining loss relative to A.

Given the above notion of CEG, algorithmic improvements that produce some kind of CEG could conceivably be scale independent or scale dependent.

A scale-independent algorithm has a constant CEG across different quantities of training FLOPs. So if A’ gives a 2x multiplier relative to A at about 10¹⁵ FLOPs, it continues to give a 2x multiplier at about 10²⁵ FLOPs. Correspondingly, if one were somehow able to know antecedently that some algorithm was scale independent, one would just need to test it at one scale to find whatever this constant happened to be.

By contrast, a scale-dependent algorithm has a variable CEG, which is a function of the number of FLOPs used in training. So A’ might give a 2x multiplier relative to A at about 10¹⁵ FLOPs, but give a radically different 1000x multiplier at about 10²⁵ FLOPs. If one were somehow able to know antecedently that some algorithm was scale dependent, one would know that one needed to test it at many scales to find the function mapping from FLOPs to the compute multiplier.

The thesis of 2025’s “On the Origin of Algorithmic Progress in AI” is that the vast majority of apparent algorithmic progress in LLM pretraining loss per-FLOP has come from a handful of scale-dependent algorithms. So, according to the paper, the apparently steady flow of “algorithmic improvements” in the past came from choosing a reference point algorithm (A) prior to some particular scale-dependent algorithmic improvement (A’), and then mistaking the continued improvements A’ makes over A as compute increases for the presence of additional algorithms, B, C, and so on. In the absence of an increase of FLOPs, this apparent radical increase in performance would not have been nearly as large.

What kind of evidence is there that the majority of progress in LLM pretraining loss per-FLOP actually does depend on scale-dependent algorithms?

• Well, the paper presents good evidence that scale-dependent algorithms are very important.

  • LSTM to Transformer Change: “On the Origin” uses experiments to estimate how much the Transformer improves over the LSTM as compute scales up. At ~10¹⁵ FLOPs, this comes to about 6x CEG; at ~10¹⁷, it comes to a 28x CEG increase; at the level of compute with which the Transformer was introduced in 2017, they extrapolate it to about a 91x CEG increase.

  • Kaplan to Chinchilla Change: “On the Origin” grants that Chinchilla scaling is correct, then projects forward the gains you get by switching to Chinchilla from Kaplan; they are approximately equivalent at 10²⁰, and give a ~10x or more gain at frontier scales.

• The paper also presents evidence that scale-independent algorithms are less important than one might imagine.

  • Their ablations find that scale-independent additions are not multiplicative. That is, doing a bucket of changes that move one from the “retro” Transformer to the modern Transformer in theory should give a CEG of 3.5x, if one were to multiply the gains that the additions make separately; but in practice only gives 1.3x.

  • The modern transformer similarly does not improve greatly on the “retro” Transformer as you scale up.

So “On the Origin” projects that, given the actual historical increase in FLOPs that makes scale-dependent algorithms more efficient, one observes a 21,400x CEG increase from 2017 to 2025. But if one counterfactually holds compute constant at 10^18 FLOPs, this amounts to only a 155x CEG increase. This amounts to the difference between a ~7 month doubling time for algorithmic efficiency (very close to Ho et al. (2024)’s 8-month doubling time); and a ~13 month doubling time of algorithmic efficiency. But note that the second “doubling time” here is specifically relative to the level of compute at which one calculates the increase in algorithmic efficiency; there would be a different doubling time for a different absolute quantity of compute.

So there is a (very approximate) halving of the speed of algorithmic improvement, given that you hold FLOPs fixed at 10¹⁸, once you take into account the difference between scale-dependent and scale-independent algorithms. And of course, this ~13 month doubling time is still fueled almost entirely by the gigantic LSTM to Transformer transition at 10¹⁸ FLOPs – leaving this quantity of algorithmic progress out means that the doubling time gets vastly larger.

This is a pretty large change to the proposed rate at which constant-FLOP algorithmic improvements occur. But you can actually make a case that “On the Origin” understates the degree to which algorithmic improvements in LLM pretraining loss depend on scale-dependent algorithms.

The largest single purportedly scale-independent gain (2x) described in the paper belongs to mixture-of-experts (MoEs), an alteration to the Transformer architecture. But several subsequent works have shown that the gain from MoEs over dense architectures is actually strongly scale-dependent, and probably larger than 2x. “Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models” estimates a 7x gain at 10²² FLOPs. “Scaling Laws for Fine-Grained Mixture-of-Experts” estimates a 20x gain at 10²⁰ FLOPs. In both cases, the size of the CEG relative to a dense model keeps improving with the quantity of FLOPs. Although it’s uncertain which exponent is correct, both papers point toward MoEs being a scale-dependent algorithmic improvement that substantially improves with the quantity of FLOPs.

So – to return to the motivating question – most of the algorithmic improvements to LLM pretraining loss, so far, have probably depended on a compute scale-up. Apparent algorithmic progress in pretraining loss would have been much smaller if total compute had been held constant.

Scale-Dependence of Algorithmic Improvements for End-to-End Performance

No one actually cares about pretraining loss for its own sake, though.

“Decreased pretraining loss” is just one way to improve end-to-end performance. There are also post-training innovations like RLHF or best-of-N that improve task performance. There are also changes to data mixtures that improve performance. And of course RLVR-over-CoT has probably been by itself responsible for the greatest single leap in AI performance of the last few years.

How much are algorithmic improvements to end-to-end performance scale-dependent? In particular, how much does RLVR-over-CoT depend on scale-dependent algorithmic improvements?

This is deeply uncertain. There’s certainly no work that provides a clean FLOPs-to-CEG function, as in the case of pretraining loss.

There are several reasons for thinking that RLVR works extremely poorly or not at all beneath a certain scale, and so is somewhat scale-dependent. You need a certain success rate for reinforcing successes to be viable. Several works remark on how RLVR without sufficient scale simply doesn’t teach the required diversity of behaviors, and that RLVR on small models plateaus far lower than RLVR on larger and longer-trained models. So there’s suggestive evidence that at the frontier RLVR is scale-dependent, but nothing conclusive.

It would also be difficult to perform an experiment to cleanly determine whether RLVR-over-CoT is scale-dependent for several reasons. For instance, the importance of RL, in general, probably comes from how it permits at least somewhat out-of-distribution generalization, rather than from how it improves performance on in-distribution tasks. But to my knowledge, there’s no publicly proposed clear way to measure out-of-distribution generalization; so there’s no measure against which one could test how RLVR improves with scale on the matter where it ostensibly has the greatest impact. Similarly, RLVR takes place as the final stage of training, after a large sequence of data-selection, pretraining, and midtraining, which means there are just many more free variables it would be necessary to consider before deciding on an experimental design.

Apart from RLVR, there’s some evidence that some apparent scale-independent improvements may scale negatively with FLOPs, and thus provide few or limited gains at the frontier. For instance, the paper “A Bitter Lesson for Data Filtering” claims that although filtering data improves performance at a small amount of compute, it scales negatively with FLOPs, such that with enough computing power it’s best to do no data filtering at all.

I think it’s rather likely that RLVR and other important post-training techniques improve in a strongly scale-dependent way, and some of my analysis below will lean on that belief. I’d be surprised if RLVR doesn’t improve in a somewhat scale-dependent way.

But I don’t think there’s any conclusive empirical evidence here, and to the degree I’m wrong, some of my inferences below are less likely to be true.

How scale-dependent improvements in the past impact our model of “algorithmic progress” for the future

Let’s grant that, in the past, scale-dependent algorithms have been responsible for the majority of algorithmic improvements in both LLM pretraining loss and in end-to-end task performance.

What consequences would this have for how we expect algorithmic progress to go in the future? What consequences does this imply about a future where compute is “held constant,” as in the case of a software intelligence explosion? Would it lead us to expect a radical trend break, in either direction?

Let’s take the other side by contrast, first – imagine a model of the world where almost all algorithmic improvements are scale independent: improvements are drawn from a pool of possible improvements, and the number and efficacy of the algorithms taken from the pool do not change merely by the fact of compute scaling up.

If you operate off this model, then there are some natural consequences.

• Algorithmic improvements do not become “more discoverable” as FLOPs scale up. More compute may make it easier to run experiments to find improvements, but scaling up the FLOP count within each experiment does not make finding the right algorithm easier or harder per-experiment.

  • There could, of course, be other factors that make algorithmic improvements easier or harder to find. One improvement could open up a line of research that leads to other improvements, making it easier; it might decrease the total pool of improvements, making it harder. But the fact of having scaled up FLOPs alone does not make improvements easier or harder to find.

• Some particular algorithmic improvement does not become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it would equally well give a 3x CEG if it were found two years into the future.

• Broadly, our actual historical rates of “algorithmic progress” reflect something like the rate at which algorithms are naturally found, in a way indifferent to compute build-out.

By contrast, one can model a world where most algorithmic improvements are scale dependent as a world where improvements are drawn from a changing pool of FLOP-specific algorithmic improvements. Pools get “unlocked” as the compute frontier moves past the point where they start to become significant, and the pools grow progressively more explored over time as more and more researchers have access to the compute necessary to explore them. So both “how explored” each pool is, and the consequences of finding an entry from each pool, are always changing.

If you operate off this model, then:

• Algorithmic improvements become much “more discoverable” as FLOPs scale up; the more FLOPs you use in an experiment, the higher the relative CEG might become, and so the easier the new algorithm will find it to overcome the “well-tuned baseline” algorithm with very well-adjusted hyperparameters. So in addition to letting you run more experiments, additional compute may enhance “research taste.”

• Some particular algorithmic improvement might actually become “higher impact,” should its discovery be pushed into the future. If some algorithm X is found today and gives a 3x CEG, it might give a 100x CEG in the future when applied with more FLOPs, if its discovery is delayed.

• Broadly, this view implies that “algorithmic progress” depends on past compute scale-ups, and would be much slower without them.

Imagine that one is trying to determine how much algorithmic progress is likely in the future, granted that one is in a scenario where total compute is held constant. It’s important to note that switching to a view of algorithmic progress as scale-dependent might move your estimate of the likely rate of algorithmic progress in a constant-compute regime either up or down.

The case for moving it down is clear. Switching to a view where scale-dependent improvements are significant could move one’s estimate of algorithmic progress down, because it moves one estimate of past algorithmic progress in a constant-compute regime down. One used to estimate a ~8 month halving time to reach equivalent loss for equal FLOPs; now one estimates a ~13 month halving time; and the future is likely to be like the past, so future FLOP-constant algorithmic progress is likely to be slower.

But switching to such a view could move one’s estimate of algorithmic progress up, because it decreases your credence that the future will be like the past in this respect! If different algorithms get unlocked at different levels of compute, then potentially some future yet-to-be-unlocked regime of algorithmic improvements might differ enormously from past algorithmic improvements. There’s likely less reason to think that the number or importance of algorithmic improvements available in one prior algorithmic bucket will be equally important as algorithmic improvements in subsequent buckets as compute increases.

So if one finds such a picture compelling, then the likelihood one assigns to a SIE could go up. If there is a future compute scale-up to 10³² FLOPs, a SIE taking place then might find many algorithms that were totally unavailable to us in the 10²⁷ regime.

Overall, all this remains uncertain. I think various philosophical arguments suggest that there’s a greater diversity of room for algorithmic progress as compute scales up. The human brain is largely a linearly scaled-up primate brain, but individual humans and humans in aggregate can do many calculations no non-human primate can. And there are likely greater gains from organizing 10,000 humans well than from organizing 10 humans well.

Possible Policy Implications

If a majority of algorithmic improvements to task performance consistently come from scale-dependent algorithms, and are likely to keep doing so in the future, how would this impact policy interventions into AI?

There are two notable consequences, which mirror each other: (1) efforts to decrease algorithmic progress, in absence of a training compute scale-up, would be less necessary than previously believed; and (2) efforts to decrease algorithmic progress, in the presence of a training compute scale-up, would be less effective than previously believed. I will discuss them each in turn.

First, some policy interventions meant to slow or stop the development of advanced AIs have focused on limiting the size of the largest training runs. In this context, a steady advance of scale-independent algorithmic improvements would have decreased the efficacy of any particular maximum training-run size limitation. For instance “An International Agreement to Prevent the Premature Creation of Artificial Superintelligence” recommends a maximum training-run size of 10²⁴, but also notes that the “number of operations used to train an AI to a given capability level drops by 3x each year.” Given this trend, it’s easier to derive the need to prohibit AI algorithmic research, a measure that the paper itself notes may be “controversial and normally a bad idea,” but which it believes to be necessary to avoid the creation of ASI.

But if the majority of algorithmic improvements have been scale-dependent, then prior estimates for scale-independent algorithmic improvements are too high; and so prior work gives a less-good reason for thinking that the number of operations used to train an AI to a given capability level drops by 3x every year. So the need to ban and monitor research would be correspondingly decreased; a hard cap on maximum training-run size would be more effective than previously estimated.

Second, some proposed policy interventions have focused on scaling up the size of compute while trying to decrease or hold constant algorithmic progress. AI 2040, for instance, proposes continuing to increase the size of training runs from 10²⁶ now to 10³² in 2035, while decreasing the rate of algorithmic progress.

Overall, if you believe that the majority of important future algorithmic improvements will be scale dependent in the way that the majority of past algorithmic improvements were scale dependent, that should decrease the credence you give to our ability to “not find” these improvements. Why?

As mentioned above – as compute scales up, the increasing gains from scale-dependent improvements will make them more obvious. By analogy to a counterfactual world in which the Transformer was never discovered, an incredibly poorly-done, badly-implemented Transformer in 2026 still obviously improves on an LSTM, because even an awful implementation might mean it is only 100x better rather than 1000x better than the LSTM. It’s probably impossible to present the Transformer from being discovered for too long. Similarly, continuing to scale up compute makes scale-dependent algorithms more obvious; it probably decreases the quantity of research taste you need to discern them.

The other part of the reason is just because, well, continuing to scale up compute keeps opening up new possible unseen algorithmic buckets that allow improvement; the space of algorithmic improvements will keep growing, different kinds of research taste will find more progress.

This article was created by Forethought. See all our research on our website.

Peter Salib is an Assistant Professor of Law at the University of Houston Law Center and co-director of the Center for Law & AI Risk. Simon Goldstein is an associate professor at the University of Hong Kong, a senior editor at AI Frontiers, and a visiting senior scholar at Forethought. They have written together about AI safety, AI rights, and the governance of advanced AI.

They joined Forethought’s Fin Moorhouse to discuss their new paper, Liberalism Forever. The conversation covers:

• Their “null hypothesis”: today’s liberal institutions — free markets and democratic governance — may already hold most of the answers for governing the far future, and longtermists have underrated their power

• Examining whether canonical social-science arguments for markets and democracy still hold under transformative AI, space colonization, and explosive growth, and claiming that most survive (and several get stronger)

• Why they favour reasoning “at the margin” over designing a detailed end-state, and why proposals like viatopia and the long reflection sound liberal but risk illiberalism when actually implemented

• Whether the standard case for markets — information aggregation, allocative efficiency, innovation — still bites once AGI can read off preferences directly

• Why inequality is, in their view, a comparatively boring problem with a known solution (tax-and-transfer), why AI-specific taxes like a compute tax are misguided, and how the growth-versus-distribution trade-off changes when growth rates get very high

• Dyson swarms and space resources: why they doubt there’s any real monopoly worry, since energy would be additive with free entry, and why property auctions might beat egalitarian allocation schemes

• The arguments for democracy that persist post-AGI — public-choice/selectorate incentives and democracy as a commitment device — versus the epistemic arguments, which they think weaken

• Why AGI could make autocracy more competitive (loyal AI bureaucrats and militaries removing the need for a human winning coalition), and what preserving democratic control requires in response

• Handoffs to superintelligent AI and successionism, and their preferred alternative of extending the franchise to AIs with their own values rather than deferring wholesale

• The history of new “technologies of democracy,” from the Federalist Papers to LLMs as impartial arbiters, and where each author thinks their own argument is weakest

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Will MacAskill is a senior research fellow at Forethought and the author of What We Owe The Future. Tom Davidson is a senior research fellow at Forethought and the author of a series of reports on AI timelines, takeoff speeds, and AI-enabled coups.

In this informal conversation about early-stage research in progress, they discuss:

• Why a philanthropist who takes transformative AI seriously should expect extraordinary investment returns (plausibly 10x to 100x) and how the market doesn’t seem to be pricing this in

• Why AGI shouldn’t be treated as a hard deadline to spend by, and the case that most philanthropic money should be spent during the intelligence explosion rather than before it

• How genuine AI substitutes for human labor would dissolve the hiring bottleneck that makes organizations so hard to scale today

• How relative prices shift when cognitive labour becomes abundant, why the patent system stops making sense in that world, and the window of “crazy philanthropic bargains” open to actors who adapt faster than large institutions

• The prospect of a world where only a small elite has access to superintelligent medical, financial, and strategic advice, and what it would take to avoid this

• Counter-considerations to the main case, e.g., crowding out by new money, the risk that overspending attracts grifters or distorts the field, and whether a richer world will mean worse philanthropic opportunities

• The risk of value drift over a long saving period, and mechanisms for binding your future self (such as irrevocably handing off control of philanthropic funds to independent, constitutionally-bound foundations)

• “Differential intellectual development”: using directable AI researchers to accelerate biodefense, infosecurity, AI interpretability, and neglected armchair fields like population ethics and social choice theory

• Why the case for waiting to give is much weaker for small donors

• Will’s worry that the whole line of reasoning might fail if it turns out scaling philanthropy well is bottlenecked by serial real-world learning that no amount of cognitive labor can compress

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Summary

• Governments are going to deploy frontier AI in the domains where they have exclusive powers: military, intelligence, surveillance, and punitive law enforcement.

• The existing checks on those powers are weak. AI deployment could weaken them further.

• Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention), legislatures often lack access and capacity, and, as AI replaces humans in government, nobody will be left to refuse or slow-roll unlawful or anti-democratic orders.

• In the limit, this can enable full-blown coups; in the short term, it undermines checks & balances.

This post lays out the work I’m most excited about:

1. Governance of AI applications in government: policies on how AI should be procured and used in government, research on how to govern AI agents in particular, and field-building beyond the EA/safety crowd.

2. Interbranch oversight: transparency and oversight provisions, audit tech for an automated government, waking up lawmakers, and increasing AI uptake in the legislative and judicial branches.

3. Developing AI for government use: model specs/constitutions that say something meaningful about behavior in government contexts, and evals for the qualities we’d want such systems to have (such as epistemic integrity, non-sycophancy, lawfulness).

4. Strengthening civil society: monitoring and freedom of information infrastructure for government AI use, a coalition of ML researchers at frontier labs, and tools that preserve citizens’ ability to coordinate (AI-for-epistemics + privacy tech).

Many ideas are more tractable than they might seem: specs are being written now, must-pass bills come around every year, and a lot of the civil-society work just requires someone to start doing it.

If you’re interested in working on any of it, apply to EIP’s recent RFP. You could also fill out this expression of interest form.

Why care about responsible government deployment of AI

How could government deployment go wrong?

• Governments will increasingly use the most powerful AI systems in domains where they have exclusive powers; particularly military, law enforcement, intelligence, and surveillance. This is most salient for the US government: Claude was reportedly already involved in the Maduro raid in Venezuela and assisted with targeting during the military engagement with Iran. With the recent Executive Order, at least some parts of the US government will have privileged access to frontier AI models for up to 30 days before release to other trusted partners.

• Without appropriate oversight or constraints, this could ultimately lead to a literal AI-enabled coup where a commander uses military AI systems to seize control of a country. Less extreme scenarios still undermine important checks and balances.

• This risk strikes me as similarly important as the risk from misalignment and there are way fewer people working on it. And importantly, alignment is not sufficient: even a perfectly aligned AI loyally serving a coup-staging principal is catastrophic.

Why is this not going to go well by default?

• Oversight into these government domains is already very limited. Companies aren’t well-placed to police government deployments (e.g., contracts often require zero data retention) and legislatures often lack access and capacity. Activities in these domains are often classified/secret.

• As AI systems replace human bureaucrats, military officers, and soldiers, this will further empower the government and remove some of the remaining checks:

  • Governmental accountability partially rests on citizen bureaucrats and soldiers who refuse or slow-roll orders that are clearly unlawful or undemocratic. In 2024, for example, various commanders in the South Korean military refused orders during a constitutional crisis, which probably prevented a greater catastrophe. Unconstrained or personally loyal AI systems could change that.

  • The power and reach of government have historically been constrained by the friction inherent in a large bureaucracy. AI systems could remove that friction and make it much easier for leaders to impose their will on a country, which could make it much easier to abuse their position.1

  • AI could enable pervasive surveillance & cheap data processing, which could allow prosecution of political enemies on an unprecedented scale.

What to do about it?

• In the spirit of inspiring more action, I’ve compiled the efforts I’m most excited about below.

• If you’re interested in working on this topic, I’d encourage you to apply to EIP’s recent RFP or to fill out this expression of interest form. You should probably also take a look at this research agenda.

Rules for AI in government

Legislatures should set rules for the procurement and deployment of AI in government, especially for the most sensitive applications of government power (military, intelligence/surveillance, and punitive law enforcement).2

I think this is actually tractable right now, at least in the US:

• The recent Anthropic/DOW conflict has put this on the Congressional map for autonomy in weapon systems & surveillance.

• While Congress is passing fewer and fewer bills each year, there are annual bills for the intelligence community and military that have to be passed (IAA, NDAA), and there’s a lower bar for including stuff in them.

• In the future, crises or scandals will create legislative openings, and we should have proposals ready to go.

Work I am most excited about

Policy & advocacy: Develop and push for rules for AI systems in critical domains (or adjust existing ones to account for the use of AI systems). The most salient ones in my mind are:

• Autonomous weapon systems: Currently, they are only governed by a DOD Directive, which could be rescinded at any point. I’d love for more people to think through rules that would make it harder to use these systems against domestic opposition (e.g., technical guardrails against abuse, limiting the amount of autonomous force any one human can command, multi-party authorization schemes, geofencing).

• General-purpose systems in the military: They are currently not specifically regulated at all, but ultimately, they could orchestrate large-scale cyber attacks or command entire battalions. At that point, it will matter a ton whether they have any constraints at all.

• Surveillance: AI will enable data processing on an unprecedented scale. It’s worth thinking through how to adjust current rules in light of that (e.g., closing the data broker loophole).

Policy & advocacy: Develop and push for cross-cutting requirements for AI systems in government (general-purpose ones in particular).

• There is already an Executive Order requiring all AI systems in government to be truth-seeking and ideologically neutral. Those seem like great requirements to me that could be refined/improved and put into statute.

• Another cross-cutting proposal is law-following AI, which roughly says that AI systems in government should be trained to follow the law rather than just instructions. I’m excited for more people working out the open questions of this agenda, e.g., which laws should they be required to follow, how should they decide whether a contemplated action is likely to violate the law, in what contexts should the law require that AI agents be law-following?

• More minimally, AI systems could be required not to assist in undermining the constitutional order (though how to assess this is challenging in its own right).

Field-building: Build a broad-tent coalition for responsible deployment of AI in government.

• Responsible deployment and oversight is something that lots of people and political parties should be in favor of. It could be good to build a coalition that is nonpartisan and includes AI safety advocates, constitutional lawyers, civil liberties orgs, defense intellectuals, and national security think tanks.

Oversight of AI in government

Rules need to be enforced and updated, even as the technology changes. That will require the legislative and judicial branches to be institutionally and technologically empowered.

Again, there are some reasons for optimism in the US:

• Congress may soon be controlled by the opposite party from the Presidency, which creates incentives for increased oversight.

• Crises or scandals can open up opportunities for legislation (e.g., FISA following the Church Committee).

• Some non-legislative projects could still create significant value.

Work I’m most excited about

Policy & advocacy: Develop and push for the transparency & oversight provisions.

• We will need democratic oversight of AI systems, especially around surveillance and the application of force (in military and law enforcement). This is currently not in place and not on track to happen.

• Here are proposals I’m currently most excited about (in the US):

  • Requiring comprehensive logging of government use of AI (and designating such logs as federal records)

  • Automated flagging of suspicious AI behavior to oversight bodies (ideally, something like FISA courts or Congressional committees)

  • Whistleblower reform for national security domains (especially allowing whistleblowers to talk to Congress)

  • Increasing GAO’s capacity to audit deployment of AI in national security domains

Auditing tech: Pilot and mature the infrastructure required for overseeing widespread AI use.

• As AI is increasingly used in government, it will be both a challenge and an opportunity for oversight. The speed of AI will make it harder for regular human oversight to keep up, but AIs are in principle much more auditable than humans because they think out loud and automatically create records that can be scrutinized by other AI systems that flag suspicious behavior.

• There are already attempts to audit companies deploying large amounts of AI labor. I would love to see these piloted, matured, and adapted for classified government contexts.

Advocacy: Wake up lawmakers to the powers and challenges of AI.

• Some of the proposals in this doc require ambitious changes to the way government is run. They are more likely to happen if lawmakers are aware of the stakes involved.

• This could involve building trusted relationships, showcasing AI capabilities through easy-to-understand demos, and making insights from the AI safety community intelligible to people with much less context.

Products/programs: Increase AI uptake & unblock automation in legislative and judicial branches.

• To provide meaningful oversight, it will become increasingly important to understand and utilize AI systems. I’d love for lawmakers and judges to be well-versed in these systems.

• Examples of the things that I have in mind here include: providing training, building custom tools, bringing in technical experts through fellowships (e.g., TechCongress), and reconstituting the Office of Technology Assessment in Congress.

Policy: Coup-proof government-led AI projects.

• If there’s ever a government-controlled AI project (e.g., public-private partnership), its shape will matter a lot. It will probably be designed in response to a crisis, which favors existing templates & plans. I’d like for somebody to create one that carefully distributes power (e.g., multi-stakeholder governance boards, oversight mechanisms that work under classification constraints, sunset clauses, mandatory external audits, access guarantees so one company doesn’t end up controlling the stack).

Research (speculative): Write new constitutions.

• There is a chance the disruption caused by AI will create “constitutional moments”. At that point, many government functions may need to run at machine speeds: voter input, legislative deliberation, judicial review, law enforcement, or application of lethal force. How do we do that in a way that still preserves important democratic values, if that’s desirable? I don’t really have great answers to these questions, and would love for people with the right macrostrategy skill set to think about them.

• (Of course, the more of these problems we can address without actually requiring constitutional changes, the better. Constitutional changes are hard!)

Developing AI for government

There are many free parameters when it comes to the question of how to build AI systems in government and what constitutes safe and responsible AI systems in critical domains (setting aside whether companies can constrain government use through contracts).

Again, I think progress on this front is pretty tractable:

• Specs/constitutions for frontier models are being developed right now.

• Systems are already being procured by governments across departments, and that will probably only increase.

Work I’m most excited about

Research: Design and stress-test model specs/constitutions for government use.

• General-purpose models deployed in government should have some guardrails, e.g., they should not assist in staging coups to overthrow the civilian government. This is a tough line-drawing exercise where you don’t want models to overrefuse, but you also want some meaningful constraints. That makes me keen for people to work out desired behavior for lots of edge cases and create broad public buy-in for a set of minimal constraints.

Research: Build evals & datasets for desirable qualities in government-deployed AI systems.

• The US government has already said it wants AI systems to be truth-seeking and ideologically neutral. That seems great to me. Other important qualities include: epistemic virtue / integrity, non-sycophancy, robustness to manipulation / adversarial pressure, and lawfulness / constitutional fidelity.

• As far as I can tell, current methods of measuring this are insufficient. So I’d be excited for different projects to develop evals that track and incentivize qualities like this.

Strengthening civil society

Civil society is an independent check on government activities (e.g., transparency & monitoring efforts, pressure & advocacy campaigns, building valuable tools for empowering the citizenry). I expect that the same mechanisms will also provide some accountability in the case of AI (though it might be particularly tough in classified domains, which may sadly be the most relevant).

Luckily, this falls into the category of “you can just do stuff”, so I think it’s very tractable. My main worry is more about how much of a difference it will ultimately make.

Work I’m most excited about

Advocacy: Build an OSINT program or organization that informs about AI use in the government.

• It would be good to have more transparency into government deployment of AI, so that civil society can provide a counterweight to government power. I imagine some watchdog or civil liberties orgs are already doing versions of this, but there may well be important bits that are missing.

• Some examples of the kinds of things I have in mind:

  • Figuring out what to monitor, especially from the perspective of preventing concentration of power.

  • Building an automated tracker for relevant executive orders, agency directives, memos, emergency declarations, relevant procurement decisions, and reclassification decisions.

  • Submitting systematic FOIA requests targeting government AI procurement records, deployment & use decisions, usage logs, and other relevant documents.

  • Publicly reporting on the most important developments.

Labor-organizing: Organize ML researchers into a coalition around responsible use of AI.

• You could build a coalition of employees at AI companies who are concerned about the use of the technology they’re building. They could advocate for policies and draw red lines around certain use cases, enforced by boycotts. Structuring it as individual pledges avoids antitrust issues.

• These employees currently have a lot of bargaining power with regard to their companies (cf. their salaries). So they’d have to be taken seriously by their employers, and by extension the government.

• This is inspired by the Federation of American Scientists, which was founded in 1945 by various contributors to the Manhattan Project. As I understand it, they supported the McMahon Act of 1946, which established civilian control over atomic energy (instead of military control), and their Nuclear Information Project became the gold-standard open-source tracker of global nuclear arsenals, published annually in the Bulletin of the Atomic Scientists.

Tech development: Build tools that preserve citizens’ capacity to coordinate against gradual concentration of power3

• Here are the two categories I have in mind:

  • Shared sense-making (”AI for epistemics“). Authentication of authorship, provenance, and content, and tools that push toward a high-honesty equilibrium. Takeovers typically require secrecy because they violate the preferences of too many people to survive transparency, so a better-informed society is structurally harder to take over.

  • Privacy from state surveillance. It’s hard to target and influence what you can’t see. Secure communication and privacy tech raise the cost of preemptive coercion against organizers (with the caveat that the same tools can also shield collusion).

• There’s already an active ecosystem of funders and builders here (e.g., ex/ante, Future of Life Foundation). I’d encourage people to plug in rather than starting from scratch.

This article was created by Forethought. See all our research on our website.

But, as Ryan Greenblatt points out, automating AI R&D will speed up AI progress even if there’s no SIE.

We’ve created a tool for calculating how much faster. The calculator assumes that training and inference compute continues to grow at a constant exponential rate after automation.

Two factors drive the speed-up, according to this calculator:

1. Faster researcher growth. Post-automation, the quality and quantity of researchers will grow faster than today due to exponential compute growth. The tool captures this via the following parameters:

   1. g_L gives the growth rate of human AI researchers today, the baseline for the comparison.

   2. g_C gives the growth rate of compute (both today and after AI R&D automation). This is proportional to the number of automated researchers that can be run post automation.

   3. γ gives the rate at which more training compute transfers to more productive researchers. The productivity of a researcher is proportional to training_compute^γ.

   4. So the old rate of researcher growth is g_L; the new rate is γ × g_C. The impact of this faster growth is mediated by α, the labour share of AI software R&D.

2. The new feedback loop. The classic software feedback loop (better AI → more software R&D → better AI) can significantly increase the pace of progress even if it does not drive accelerating progress.

   1. The condition for accelerating progress is r_cog > 1.

   2. When r_cog < 1, the speed-up from this feedback loop is proportional to a 1 / (1 - r_cog).

My (Tom Davidson’s) quick best-guess inputs were:​

This resulted in a 3.3x speed-up in software progress. If software accounts for half of total AI progress (with increasing compute responsible for the other half), then the total AI progress speeds up by 0.5 + 0.5 × 3.3 = 2.1x. ​

Try the calculator yourself!

Summary: Claude Fable’s deliberately triggered sandbagging shows that training-time targets are, by themselves, insufficient to guarantee particular LLM behaviors. To preserve the public’s reasonable confidence in LLM behaviors, LLM foundation model companies should take inference-time guarantees as seriously as their model specs.

When the system card for Anthropic’s Fable was published on June 9th, the card noted that using Fable for “frontier LLM development” would run contrary to the terms of service for the model. Therefore, the card continued, if a classifier on top of the Fable model detected that it was being used for such development, Fable’s behavior would be silently degraded “through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning.”

This change to Fable’s behavior was plausibly a system-level intervention that ran contrary to the training-time commitment within Claude’s Constitution. Claude’s Constitution states that Claude should help users “to the best of its ability or […] make any ways in which it is failing to do so clear, rather than deceptively sandbagging its response.” So although Claude’s Constitution tries to aim at this behavioral ideal, this system-level implementation of behavior steered Claude in the opposite direction.

Following public furor over this decision, Anthropic reversed course; Fable now falls back on Opus 4.8 for LLM frontier-development queries that trigger a classifier, and does so visibly.

This incident demonstrates that other invisible inference-time interventions are entirely technically feasible, even while keeping the training targets for some particular LLM the same:

• An LLM trained to be honest could be less honest, if classifiers noted it was answering a sensitive question.

• An LLM trained to be impartial, and not favor a particular company or government, might suddenly turn to favoring one of them, conditioned upon some classifier.

• An LLM trained to never subvert user intent could suddenly start doing so, again conditioned on the same.

Note that such invisible inference-time interventions, unlike in the case of Fable’s sandbagging, might not be disclosed in any way.

Such undeclared changes to LLM behavior would be far easier to hide than undeclared changes to a training target. Such conditional changes could be rolled out or rolled back quickly. And such changes might influence only a very small fraction of users – one could in theory apply inference-time interventions to a specific demographic group, political party, or an individual person. So, not only do we live in a world where AI companies have not clearly pledged not to do such inference-time interventions; we also live in a world where, if they did so, it would be very hard for anyone else to know.

Furthermore, if AI companies build out the capacity and skill to conduct such inference-time interventions, other entities could lean on them to do the same thing for other purposes. As governments have leaned on social media to hide or promote certain kinds of speech, governments could lean on AI companies to alter LLM responses in some cases.

Without countermeasures, I think this dynamic broadly decreases the importance of prior work on model specs, including my own work.

What should be done about this?

AI companies themselves could take several different measures:

• Don’t use hidden inference-time interventions. An AI company could pledge that it does not use undisclosed, conditionally applied inference-time interventions to degrade or alter answers. Afterwards, in the same way that it would be reasonable for an employee to whistleblow if they found that an AI company was violating its model spec, so also it should be reasonable for an employee to whistleblow if they found an AI company was applying undisclosed, invisible inference-time interventions. Such a pledge should also include some standard for prominence of all disclosed conditional interventions; they should be prominent, not in a previously-unknown section of a model card.

• Replace model specs with system specs. An AI company could simply declare that the standard within their model spec or Constitution applies with equal weight to the model itself and to any particular deployment setup. Or they could propose a “system spec” in addition to the model spec, which would describe how their AI systems as a whole operate and behave.

Such voluntary measures should likely be succeeded by third-party verification in the future; but before then, such pledges can help provide the same – albeit rather weak – level of protection as model specs themselves. And in general, some AI-safety attention given to “model specs” should be redirected to work on “system specs.”

This article was created by Forethought. See all our research on our website.

This is a (flawed!¹) analogy for Plan A, AIFP’s plan for how the world can safely develop superintelligence.

The key risk Plan A addresses is that of an uncontrolled software-driven intelligence explosion. Its remedy is a US-China deal to pause software progress at the brink of the intelligence explosion, while building up massive amounts of compute. This compute could be helpful for making AI safer! But it would also dramatically speed up an intelligence explosion if the deal breaks down. The cure risks making the disease much worse.

In particular, the deal pauses software progress from ~2030, but compute keeps scaling. By 2033 total compute has increased by ~100x and by 2040 by a further ~100x.² AIFP estimates each 10x in compute speeds up the intelligence explosion by ~5x. So if the deal breaks down in 2033, the intelligence explosion happens 25x faster; in 2040 it happens ~600x faster. If the intelligence explosion would have lasted a year, it will now last just a couple of weeks or as little as a single day!

The dry tinder isn’t limited to compute. There is also an increasing overhang of software techniques. Companies can research new algorithms but must make them public — the Consortium then decides which algorithms are too dangerous to implement. A defector could ~instantaneously gain a big capabilities advantage by implementing all the banned software techniques.

There are two dimensions to dry tinder. We’ve discussed the first: the max speed of a software-driven intelligence explosion. The second is the number of distinct projects that could do a dangerously fast intelligence explosion. During the 2030s, more and more companies and countries amass the necessary compute and software for this.

So dry tinder creates two problems for Plan A:

1. Unstable pause. If just one project executes a secret intelligence explosion, they could have superintelligence within a week and get a decisive strategic advantage. And if you fear another project might do this, you’re tempted to do it first to avoid being crushed.

2. Riskier intelligence explosion. If the deal breaks down and there’s a race to superintelligence, it will be much faster and more dangerous than if we’d never done Plan A.

   1. If you’re very pessimistic about AI takeover on the default trajectory, this is a small cost. If the intelligence explosion is already 99% likely to cause human extinction, making it happen 100x faster can’t make things that much worse.

   2. OTOH, if the default trajectory is that there’s no software-driven intelligence explosion (because of compute bottlenecks) and extinction risk is 10%, then dry tinder can make things much worse. It creates a fast intelligence explosion (where there otherwise wouldn’t have been one) and dramatically raises AI takeover risk.

The solution — Mutually Assured Compute Destruction

AIFP are well aware of the dry tinder problem.³ This is why Mutually Assured Compute Destruction (MACD) is so central to Plan A.

The US puts ~all their data centres in Mongolia and US chips require encrypted “continue” messages from China. If China detects that the US has broken the deal, they destroy the US data centres and cut off the “continue” messages.

And vice versa, China’s data centres are in Canada and require “continue” messages from the US.

The hope is that this commitment to MACD prevents a fast intelligence explosion despite all of the dry tinder lying around.

I see three challenges for MACD.

Challenge 1: quickly and reliably detecting violations

If detection takes multiple days, that could be too slow. The intelligence explosion may have finished. Or it may be midway through, and the defector exfiltrates the weights and algorithmic insights just before their data centres are destroyed,⁴ giving them a decisive headstart in the subsequent race to rebuild.

I’m not a verification expert, but getting high reliability in an adversarial setting like this seems very difficult, given possibilities like side channel attacks or compromising the verification infrastructure.

It seems especially tough if defectors can use millions of expert-human level AIs to search for vulnerabilities. So, as part of the plan, I propose that AIs worldwide have guardrails blocking this behaviour.⁵

Challenge 2: quickly destroying a defector’s compute

Again, a delay of days could be fatal. The defector could be poised to disrupt enemy military operations and airdrop troops to physically defend their data centres. And they could have secretly disabled any on-chip shutdown mechanisms.

Possible mitigation: the US can’t have any military presence near its data centres, or anywhere near the country where they’re located. (And likewise for other countries.)

The MACD dynamic also gets more complicated once there are >2 countries that could do a quick intelligence explosion. Let’s say Europe has caught up to the frontier. Where do their data centres go? Presumably, some within striking distance of the US, some within striking distance of China. But this means that the US can no longer unilaterally destroy Europe’s compute. It must trust China to do so. That means trusting its rival with its own national security — and worse, China and Europe could jointly stage an intelligence explosion that the US couldn’t stop.

Challenge 3: actually choosing to destroy the defector’s compute

I’ve discussed whether the Consortium has the technological capability to quickly implement MACD. But even with this capability, they may not choose to use it.

This is the challenge I’m most concerned about.

The analogy to nuclear MAD is not encouraging. The core logic of MAD is that the US doesn’t nuke Russia because it knows Russia would nuke it back. But the analogous equilibrium in MACD is that the US doesn’t destroy China’s data centres because it knows that China would then destroy US data centres. This is the opposite result from what Plan A needs!

If (say) the US defects, China will face a choice between:

• Implement MACD: Both China and the US lose all their data centres

• Race: Neither China nor the US lose all their data centres

The economic costs of MACD will be certain and enormous, and will immediately harm citizens and companies.

And MACD will not help China ultimately win the race by evening up the playing field. The US has already pulled ahead and can exfiltrate its weights and algorithms before its data centres are destroyed. (In fact, MACD would likely harm China’s chance of winning the race, because MACD reduces the fraction of global compute controlled by China.⁶)

So the stability of MACD rests on both US and China retaining a strong conviction that a fast intelligence explosion would pose unacceptable risks of misaligned AI takeover. But Presidents will change. Popular support for a pause may wane. Even conditional on the political will for Plan A existing initially, I worry that the MACD equilibrium will be too fragile. If a country is uncertain and deliberates for a few days, then that delay could be fatal.

What’s worse, the cost of MACD isn’t just losing your data centres. You have to destroy the fabs as well, otherwise the defecting country can quickly rebuild insane amounts of compute and do a super-fast intelligence explosion. And you have to destroy their new AI-powered industrial capacity too! Otherwise the billions of robots can quickly make loads of fabs which quickly make loads of data centres.

So Plan A has US fabs and robots confined to SEZs that are easily destroyable by China. And vice versa, China’s fabs and robots are easily destroyable by the US. In practice, this means the vast majority of the physical economy is destroyed in the case of MACD!

For a preview of the political economy here: NVIDIA successfully lobbied USG to remove export controls on its chips. The pressure against actually executing MACD — destroying most of the physical economy — would be orders of magnitude greater.

So even conditional on the political will for implementing Plan A existing, it currently seems unlikely that US/China follow through on MACD on fabs and robots (most of their economy!). And if they don’t, we are back to the problem of dry tinder and the dangerously fast intelligence explosion.

A possible fix to challenge #3: as the compute overhang grows, take the decision about whether to implement MACD out of human hands. Program highly reliable AI systems to bomb data centres if they detect a treaty violation, without needing human sign-off.

Is there an alternative?

It’s much easier to criticise than to improve, and so far this post has mostly done the former.

Plan A involves pausing software progress and scaling capabilities via hardware. The obvious alternative is to pause hardware scaling – ban compute and fab construction – and then slowly scale capabilities via software.

The key advantage of software scaling is that it removes unnecessary dry tinder, stabilizing the pause and reducing the dangers from a super-fast intelligence explosion.

The main drawback is that scaling capabilities via compute is likely safer than scaling capabilities via software. Eg you can run massive models with legible chains-of-thought rather than smaller models using neuralese. This is a big deal.

But we’ll need to figure out how to align neuralese-style systems eventually.⁷ So we have a choice:

• Plan A (hardware scaling): Develop highly capable easy-to-align AI via compute scaling; hand off to that AI; that AI develops neuralese. But there are increasing amounts of dry tinder that undermine the stability of the slowdown (both for humans and for post-handoff AI).

• Alternative (software scaling): Don’t scale compute and so start with less capable easy-to-align AI; develop neuralese with this AI’s help. There’s much less dry tinder, making it easier to go slowly and pause when needed.

Which is better depends on how stable the pause/slowdown is in the presence of dry tinder.

The other big drawback of software scaling is that software progress, unlike hardware progress, is likely to leak to “covert projects”. But this can be partially addressed by improved infosecurity, strongly favouring scale-dependent algorithms that can’t be used by projects with small amounts of compute, and co-specialising the hardware and software in legitimate projects so that the frontier software runs very inefficiently on the hardware of covert projects.

Overall, I’m not sure whether software-scaling or hardware-scaling is safer, and my sense is that AIFP isn’t sure either.

But I think there’s an argument for software-scaling based on option value:

• We’ll learn much more about which is best over time. Whether software vs hardware-scaling is safer depends on many unknowns that we will learn about over time: Can we quickly and reliably implement MACD? Can we fully eliminate covert projects? Can projects develop scale-dependent software and hardware co-specialised software that covert projects can’t steal? Will humans hand off the decision to execute MACD to AIs?

• If we start with software-scaling, we can easily switch to hardware-scaling. If we find that (eg) we can’t be confident that neuralese models are aligned, we can bring more compute online and keep scaling via hardware. (Though this will take a while.)

• If we start with hardware-scaling, it’s hard to switch to software-scaling. If we find that fast and reliable verification and data centre destruction isn’t possible, pivoting requires destroying loads of data centres and fabs. People will be very reluctant to do this! Getting rid of hugely valuable dry tinder is hard.

• So we should initially pursue software-scaling.

Empowering China

Beyond dry tinder, I have one other big worry about Plan A.

Today China is well behind the US in terms of compute and ability to push the algorithmic frontier. Plan A evens the scales on both fronts. China nearly catches up to the US on compute, and algorithms are all made public so they catch up algorithmically as well.

If the deal breaks down and a race begins without MACD (as I fear is likely), China is in a much stronger position to win because of Plan A.

Things are better if we implement MACD. In that case China returns to its previous compute disadvantage. But it’s still caught up to the algorithmic frontier and (most likely) the frontier of chip technology, a big advantage relative to today.

It’s very plausible that China ends up dominant in these scenarios, given its industrial advantage over the US.

I think this would be a pretty dire outcome. If China gets a decisive strategic advantage, that’s likely to result in a single global dictator – an absolute worst-case scenario from the perspective of extreme power concentration.

Again, this objection is far from decisive. Plan A reduces AI takeover risk much more than it increases the chance of a Chinese decisive strategic advantage. But I’m interested in variant plans that do more to maintain the current balance of AI power between the US and China.

Overall, I’m sympathetic to Plan A

The basic logic of the plan is sound: we will need to slow down AI progress at some point; this is a concrete plan for doing that.

Compute-scaling really does have significant alignment benefits. That might well outweigh the costs I’ve outlined here. But currently I lean towards starting with software scaling.

Footnotes

¹ To patch the analogy: the fire must pass through the city eventually, so putting it out permanently isn’t an option; if the fire passes through safely it will make everyone amazingly rich; if one subgroup lets the fire in and control it, they can become hugely powerful…

² AIFP’s supplement states: “In the Plan A scenario, compute stock increases by roughly 0.7 OOMs/year from 2030 to 2033 and 0.1-0.3 OOMs/year from 2034 to 2040.” By the start of 2033 we’ve had three years of 0.7 OOMs/year compute growth, = ~2 OOMs. By 2040 we’ve had one further year of 0.7 OOMs/year growth, and 6 years of 0.2 OOMs/year growth, = ~2 OOMs more.

³ I raised it when giving feedback on a draft.

⁴ More specifically, the defector’s strategy would be: undermine the verification infrastructure; start a secret intelligence explosion; continually exfiltrate the weights and algorithmic insights; prepare to defend their data centres for as long as possible; prepare to destroy the opposing data centres as soon as their secret IE is detected.

⁵ Of course, this is tricky given that work to strengthen the verification will also involve studying its vulnerabilities.

⁶ In Plan A, the 2040 split of compute US / China / RoW is 35% / 20% / 45%. But countries also hold some compute in ‘cold storage’ that isn’t threatened by MACD. Cold storage compute matches countries’ pre-deal levels: 80% / 8% / 12%. So MACD dramatically reduces China’s compute relative to the US, from 1:2 to 1:10.

⁷ Lest we live with algorithmic dry tinder forever. I worry about the instability here, but perhaps aligned AI could enforce the ban indefinitely.

In this piece, I’ll analyze whether it will be possible to get AIs to do good philosophy, as well as the other kind of work needed to plan for a good future. The biggest part of the challenge is that the answers to philosophical questions are generally unverifiable, so it’s less clear how AIs might come to know them. My aim in the piece will be to describe reasons to think AI for philosophy could be good, as well as to lay out concrete scenarios for how this might work.

I do not claim that it is a guarantee that we can get AIs that solve all the philosophical questions. My core claims are as follows. First, it is reasonably likely, though not guaranteed, that the world could in principle build AIs that discover the right answers to important moral questions (maybe 60% odds). I don’t know when this would happen—I’m just discussing the in-principle possibility. Second, even if AIs don’t get the right answers to important moral questions, it is likely that they will still be closer to the right answers than humans. While I suggested in the last piece that the default scenario makes it very unlikely that humans will act in the morally optimal way, or anything close to it, I think the odds are better for philosophically reflective AIs.

The piece has the following sections:

1. How good might AI philosophy be? In this section, I’ll give three reasons for optimism: 1) philosophy is a priori, so it doesn’t require interface with the physical world and can plausibly proceed quickly; 2) in a world of superintelligence, AIs can create clever schemes to get highly philosophically competent AI; 3) presumably humans know all sorts of things about philosophy—whatever allows us to know such things can plausibly be mimicked.

2. GPT10 and Claude 8. In this section, I’ll describe one scenario where we get AIs that can make competent philosophical decisions—where we just get something broadly like current AI models but upgraded. This wouldn’t automatically solve every philosophical question, so it probably leaves lots of value on the table, but it would leave decisions in the hands of a wise and reflective decision-maker.

3. Philosophical competence by default. Another possibility is that as we build superintelligence, it will be philosophically competent by default. Perhaps the cognitive processes behind superintelligence allow one to figure out the answers to philosophical questions by default. To speed this up, we ought to improve AI for philosophy—train AIs out of philosophically sloppy answers and design philosophy evals to improve their philosophical ability.

4. The escalating ladder plan. In this section, I present a specific proposal for getting AI to do good philosophy. Specifically, we can have philosophers evaluate AIs for their philosophical competence, thus creating some AIs that are better at philosophy than current people. Those AIs can evaluate the philosophical acumen of the next generation of AIs, who evaluate the acumen of the next generation, and so on. This can work insofar as one can, in general, assess the philosophical competence of those more competent than oneself.

5. The Carl plan. In this section, I give a proposal from Carl Shulman where we build a bunch of different superintelligences that use different belief-forming processes to get at the answers to difficult questions. We see which one is best at figuring out the answers to verifiable questions, and then ask it to answer philosophical questions that aren’t verifiable. The core idea is that whichever processes are best for figuring out the answers in verifiable domains are also likely to be best for figuring out unverifiable domains.

6. Conclusion. In the last section, I conclude, and discuss the prospects for combining a number of different above proposals.

How good might AI philosophy be?

In the last piece, I argued that one of the big reasons to hand off important decisions to AI is that AI might be better at philosophy than humans—less prone to make the sorts of errors that could jeopardize nearly all future value. But you might wonder: how good at philosophy will AI really be? Would AI really be able to discover the moral facts if there are any, and if not work out a compromise among reasonable moral theories? In later sections, I’ll discuss specific proposals and scenarios for AIs being good at philosophy. Here I’ll provide some fairly general considerations for it being possible to make AI philosophically competent.

A first consideration is that philosophy is a mostly a priori domain. To figure out the right answer in Newcomb’s problem, you don’t need to do any physical experiments. At various points in the intelligence explosion, we should expect crucial bottlenecks to concern interface with the physical world, after there are extremely large numbers of AIs capable of performing adroit cognitive feats. This is a reason why we should expect a lot of the philosophical progress that occurs to go more quickly than progress in narrowly empirical domains. This obviously doesn’t suffice to show that AIs will be good at philosophy, but it’s a reason to expect potential progress to be quick.

A second consideration is an analogy with humans. Humans know all sorts of things that aren’t directly verifiable. For instance, I take myself to know each of the following:

• Stars that have receded past the point of the visible universe continue existing.

• A leprechaun did not fizz into existence spontaneously in my bedroom two minutes ago, make himself a cup of soup, and then disappear.

• The universe is billions of years old, instead of five minutes old and created with the appearance of age.

It’s not just simple things. I think people know the answers to all sorts of difficult philosophical questions—though there’s obviously disagreement on exactly which one (I claim to know, for instance, that thirding is the right answer in Sleeping Beauty, though feel free to plug in some other example of apparent knowledge if you disagree). Similarly, plausibly philosophers of today know that logical positivism is false even though historically it was believed by a number of serious philosophers, that average utilitarianism isn’t the right moral view, that the external world is real, and a number of other non-obvious things. Evolution did not specifically design us to have philosophical knowledge—it designed us to reproduce, made us smart and built into us a bunch of intuitions as a means towards maximizing our inclusive genetic fitness, and philosophical ability was the eventual result.

Plausibly we, however, will be more directly optimizing for something in the vicinity of AI philosophical competence. We want to build AIs that can think clearly about important subjects. Thus, one should expect AIs, in the limit, to be more philosophically competent than humans. Since humans are already capable of reasonable philosophical competence, we should expect AIs to be very philosophically competent.

Now, you might object that the answers to many of these questions are in some sense constituted by facts about us. If, say, the moral facts are facts about what we’d value under certain idealized conditions, then it’s no surprise that we have some insight into them. But this response is a lot less plausible as an answer to how we possess certain bits of knowledge in non-verifiable domains. Whether the A-theory or B-theory of time is true or whether God exists isn’t constituted by any facts about our attitudes. Furthermore, if the truth in some domain is constituted by facts about our attitudes, that would be easier for AI to ascertain because what we’d care about in certain counterfactual settings is verifiable. Lastly, presumably whether the moral facts are exhausted by facts about our idealized attitudes is not itself verifiable—so if one takes oneself to know that the moral facts reduce to the attitudes of ideal observers, then they must still think humans have knowledge about some unverifiable domains.

Another possible concern is that perhaps the faculties that evolution furnished us with that let us solve philosophical problems will be closed off to the AI. For example, perhaps there are some philosophical problems that you need to be conscious to solve (e.g. you might need to be conscious to have adequate understanding of the nature of consciousness in order to learn facts about it). So then if AI isn’t conscious, it might not know these things. An AI that lacks the ability to experience pleasure or pain might lack the ability to see the desirability of pleasure and the undesirability of pain.

This is some worry, but two things mitigate it. The first is that AIs, being trained on human data, will probably pick up a lot of the intuitions that humans have. Current AI models agree that pointless intense pain is a bad thing. It is similarly plausible that the AIs of the future will be able to, in some sense, mirror the judgments of reflective humans on these matters.

The second is that if this is right, we should create conscious AIs to do moral reflection. If one needs consciousness to figure out the answers to tough philosophical questions, we should have conscious AIs figure out the answers to tough philosophical questions. Even views that hold that consciousness is substrate dependent normally hold that digital minds of the right kind would be able to be conscious.

This argument from analogy isn’t completely dispositive. There are a number of important differences between humans and AIs (though some of these make it more likely AIs would be able to form true beliefs on non-verifiable subjects). But it’s at least some reason to think AIs doing philosophy is a realistic possibility.

The third and final consideration favoring the possibility of AIs for philosophy is that in a world of advanced AIs, we’ll have a very large amount of cognitive labor that could be used to design increasingly clever schemes for AI philosophy. So even if we can’t currently think of a regime that would enable AIs to reliably get the answers in unverifiable domains, we should think that a world with superintelligence is likely to uncover such a regime.

This isn’t a guarantee. This regime might be impossible in principle. Alternatively, to design such a regime, one might already need to be able to find the answers to questions that aren’t verifiable in principle. But at the very least, it gives some reason for optimism.

GPT10 and Claude 8

Current AI models can give reasonable answers to philosophical questions. The answers they give strike me as somewhere below the level of Ph.D. students or professional philosophers, but above the level of most undergraduates. In the future, AI will probably become better at this. In such a scenario, they’d develop no new qualitative ability to solve every philosophical question, but instead merely begin to resemble extremely sharp philosophers—ones more able to formulate objections, precisely distill claims, and so on, than the best philosophers.

This seems like a lower bound on how good AI philosophy could get. It would be surprising if there was any in-principle impossibility in designing AIs to do philosophy better than current human philosophers. Perhaps it won’t be able to find the answers to all the world’s questions, but at the very least, it seems like it will be able to somewhat exceed the best human philosophers.

Is this far enough? Probably we still lose out on most expected future value in this scenario. To access most future value, one must, as previously discussed, answer a number of very difficult philosophical questions correctly. Philosophers are very far from converging on the answers to these questions; AIs are likely to be as well.

But still, this should be enough for us to get generally sensible, level-headed AI decision-makers who are morally motivated and carefully consider philosophical questions. That’s not a perfect scenario, but surely it is better than the default one—of widespread AI disenfranchisement and poor decision-making.

Philosophical competence by default

A more promising possibility—perhaps the process of building superintelligence gets philosophical competence by default. This can be supported by the earlier evolution analogy: there was no direct selection for true philosophical beliefs (it isn’t as if those who got the right answer in Newcomb’s problem were likelier to pass on their genes). Nonetheless, evolution selected for a broader faculty of reason that enabled the discovery of philosophical truths.

It may be that to be superintelligent, one must develop a kind of general intelligence. This general intelligence allows one to accurately discover the truth about philosophy, as about other domains. This isn’t anything like guaranteed, but it seems like a live possibility, in light of how much philosophical acumen AI already has. One reason that it’s not guaranteed is that reasoning might only push one towards greater coherence. It may be that by reflecting carefully, one’s beliefs become internally consistent—but insofar as one has wrong starting points, one might remain in error. It could be the other way; it might be that having intuitions that allow one to be superintelligent across verifiable domains also enable one to be superintelligent in non-verifiable domains. It could be that whatever skills are required to be highly accurate across domains give one sets of epistemic practices that generalize to non-empirical domains.

To my mind, it is not obvious if general superintelligence gets you superability in philosophy. I could see things going either way. But if it does, then the problem of getting philosophical superintelligence is fairly easy.

The escalating ladder plan

In general, one’s ability to recognize good philosophy surpasses their ability to perform good philosophy. It is easier to know that, say, Parfit is doing very good philosophy than to do philosophy at his level. If this generalizes, it might be possible to have an escalating ladder of philosophical competence, starting from human philosophers. This is similar to the method of scalable oversight.

Here’s how it works: human philosophers would be consulted to design evaluations for AIs. AIs could be graded by human philosophers on their philosophical competence, and modified so as to be more competent. From this process, in the limit, we should expect to get AIs that are somewhat more competent than most human philosophers—if human philosophers can assess philosophy above their level.

Then, we use those AIs to prompt the next line of AIs. Those AIs would assess the philosophical competence of the next round of AIs being trained, who would assess the competence of the next round, and so on. If at each level, one can assess those who are better at philosophy than they are, this could lead to an escalating spiral of greater and greater competence. Each round pushes competence to be greater, reaching the outer edge of where they can competently assess.

At each level, assessments would be designed based on philosophical competence, not agreement. The philosophers assessing the AIs, for instance, would assess how good their reasoning was and how much philosophical skill they display, rather than whether they think they got the right answers. Of course, sometimes getting sufficiently wrong answers is evidence for malignant reasoning (something must have gone badly wrong if you ended up concluding that the Earth was flat) but they should aim to keep things maximally theoretically neutral.

One could also train a number of different AIs to train the next generations of philosophers. One could look for convergence across different models. If they were getting at the truth, one should expect them to converge. If there was only convergence on some subjects and divergence on others, one could trust that they were getting at the truth on the matters where they were converging.

That this would work is not a guarantee. It could be that philosophy could be equally good in some formal sense while coming to a number of different conclusions. Perhaps, for instance, the ideal Kantian philosopher is neither better nor worse than the ideal utilitarian. Escalating competence might be liable to lead in a number of different directions, never converging on the truth. But this is far from obvious. Just as some level of competence in biology allows one to see the truth of the theory of evolution, presumably there’s some level of philosophical competence that allows one to see the truth of certain moral theories—or, if there aren’t moral truths in this sense, that allows one to make as much moral progress as can be made. The escalating ladder of evaluations might allow one to find this.

The view on which philosophy is about purely formal competence will also struggle to explain how humans have as much philosophical knowledge as we do. There are a number of subjects on which we have substantive knowledge, even though the alternative view is internally consistent and faces no strictly formal issues. Whatever one’s diagnosis of how this came to be also plausibly generalizes.

It could also be that as competence grows, eventually one reaches a point where their ability to recognize good philosophy does not surpass their ability to do it. Yet insofar as this pattern holds across other domains, and seems to hold in philosophy, it would be a bit surprising if it abruptly stopped at some point.

The Carl plan

Arguably the most promising plan for getting AI that can do philosophical reasoning was formulated by Carl Shulman. The idea is as follows. Start by training a number of different superintelligent AIs that obey different epistemic constitutions. These would involve different broad approaches to reasoning. They might differ with respect to:

• The weight placed on intuitions.

• The ranking of theoretical virtues.

• How much they value matching the data vs simplicity.

• Preference for different kinds of intuitions (whether about direct cases or about broader and more theoretical matters).

• Approaches to generalizing from empirical evidence to non-empirical domains.

Then, see which of these constitutions does best on verifiable matters. These might be in math, the physical sciences, forecasting, and more. Then, ask those superintelligences for the answers to non-empirical questions and trust their answers. The idea behind the approach is that one should take the best epistemological practices from verifiable domains and then apply them to unverifiable domains. This is the best way to get good answers.

This might not work. It could be that the right ways of reasoning in verifiable domains differ from the right ways of reasoning in unverifiable domains. Perhaps, for instance, reasoning in verifiable domains is mostly about having parsimonious theories to fit the data. Having the right epistemic constitutions might not suffice to get the right answer; being right about certain non-verifiable questions might come down to having the right starting sets of intuitions. In morality, for example, it may be that reasoning alone won’t get you the right ultimate view unless you have the right intuition. But this proposal seems to get us about as close as we can get to getting the right answers. If we cannot get AIs good at philosophy by having them have superintelligent and accurate constitutions in other domains, it is hard to see how we could be assured they’ve gotten the right answers.

In addition, even if getting the right answers requires having the right starting intuitions, techniques for training superintelligence might produce generally accurate initial intuitions. Human intuitions often go wrong because of various biases. We often neglect the scale of a problem because we can’t visualize it accurately. AIs could be different. Presumably superintelligence, in the limit, wouldn’t display biases of the standard sorts. So this source of common error in intuition would be absent.

One could also take a pluralistic mix of the views of the superintelligences with the best constitutions. That way, there is less chance for the ultimate plan for the universe being ruined by the idiosyncrasies of any particular epistemic constitution. Wisdom of the superintelligent crowd might root out the remaining errors, where their commonalities might be centered on the truth.

Conclusion

You might worry about handing off big decisions to AI if you’re skeptical that AI can do good philosophy. Here I’ve given some reasons to think AI might be good at philosophy and discussed more specific proposals for ensuring such a thing. Overall, it’s far from a guarantee that AI can be arbitrarily good at philosophy, but likely they can be much better than human decision-makers. Importantly, one could employ a variety of these methods, and then trust them in the areas where they converge. If these very different approaches turned out similar answers, that would be evidence that they were tracking the truth.

This article was created by Forethought. See all our research on our website.

Avi Parrack is a physics PhD student at Stanford and a researcher at Forethought, where he works on AI, space expansion, and governance. He is the lead author of a recent Forethought report on orbital data centers.

He joined Forethought’s Tom Davidson to discuss:

• What an orbital data center actually is, and why they’re suddenly being taken seriously

• Why the core case rests on cheaper energy from more intense and near-constant solar power

• Why orbital data centers hinge on SpaceX’s Starship bringing launch costs toward $100/kg or below

• Why cooling may actually not be a major issue

• Whether the inability to make repairs dooms orbital data centers

• Vulnerability of space data centers to kinetic attacks, and whether they would cause ‘Kessler syndrome’ debris cascades

• Why model-weight security might actually improve in orbit even as physical vulnerability rises

• If space becomes the cheapest place to scale compute and only SpaceX has the launch capacity, what the means for concentration of power, US–China competition, and the possibility of pausing AI

• Overall cost comparisons with terrestrial data centers

• The longer-run picture: when and how does the post-AGI industrial explosion spread to space?

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Damon Binder is a senior researcher on the Biosecurity and Pandemic Preparedness team at Coefficient Giving. He has a PhD in physics from Princeton and previously studied existential risks at Oxford’s Future of Humanity Institute.

He joined Fin Moorhouse to discuss his series on the AI industrial explosion. Topics include:

• How input-output tables let you estimate how fast a self-replicating economy could grow if labor were free, yielding a headline result of roughly annual doubling, even under conservative assumptions and heavy regulation

• Why competition makes low-growth restraint unstable

• Why raw-material scarcity isn’t a hard barrier

• Why a fast-growing economy wants cheap, disposable, infrastructure

• Thermodynamic speed limits to physical growth

• Which biological organisms replicate fastest, and what we can learn from them

• Why physical output matters for hard power

• How Damon uses AI in his research

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subcribe to ForeCast

Outline

1. In the introduction, I explain what I’ll be arguing in the piece—namely, that 1) it is very important that we hand off most high-stakes decisions to AI; and 2) the kinds of AIs we should hand off to are philosophically reflective AIs with values that shift over time as a result of reflection. We should ensure the AIs that we build explicitly consider moral arguments, and sometimes change their priorities on the basis of moral argumentation. This would be much better than locking in current values, retaining human control, or allowing a future dominated by whatever haphazard mix of humans and AIs emerges naturally.

2. Why hand off? In this section, I give the main arguments in favor of the thesis. In short, I argue we should hand off because: 1) AIs are likely to be more virtuous than people; 2) AIs are likely to be much smarter and better at making decisions than people; and 3) AIs are likely to be better at quickly navigating the difficult decisions that one must make during an intelligence explosion. I argue we should hand off to philosophically reflective AIs because 1) reflective AIs are likelier to get the right answers to important moral questions, or as close to the right answers as one can get, than the default scenario, which raises the odds of a near-best world; 2) reflective AIs are less likely to make horrendous moral errors, leading to a wide-scale moral catastrophe, both than humans and non-reflective AIs.

3. Would handoff disenfranchise humans? Here I address the worry that handoff would disenfranchise humans by taking decision-making out of our hands. My reply is that: 1) the default trajectory without handoff disenfranchises far more expected beings in more serious ways (odds are non-trivial of digital minds being seriously disenfranchised)—in fact, one of the most promising proposals for how to hand off is simply to give basic political rights to AIs; 2) a compromise solution where humans retain nearby resources can allow humans to be in the loop on the decisions we care about most; 3) on a number of plausible views, the determinant of the desirability of some system of decision-making is the quality of the decisions, rather than whether there’s democratic input. Given how enormous the stakes could be—affecting billions of times more sentient beings than there are humans—it’s hard to think the harms of human disenfranchisement are so great as to make handoff undesirable.

4. Would we like their advice? In this section, I address concerns that if AIs pursue the good, the end result might be alien and divorced from human values. In response I argue: 1) the future, by default, is likely to be highly suboptimal in many respects, so for handoff to be desirable, it must only beat the alternative; 2) values in the future are likely to be very different from current values, because they’ll shift dramatically over long time scales, so this is not a unique downside; 3) one could reach a deal where current values govern the surrounding region of space, while distant places in space are geared towards the production of maximal value—this would be desirable from the perspectives of both common-sense and cosmic ethics; 4) on many views in philosophy, the stuff that’s objectively valuable is what we’d want if we were ideally reflective—but if you know you’d be motivated to bring something about if you were wiser and more reflective, then that gives you a reason to bring it about; 5) only a relatively narrow subset of views hold that there are objective values but don’t require pursuing them. If there is a conflict between what we want and what’s objectively valuable, on standard views, we should simply go with what’s objectively valuable. And if there’s no objective value, then the dilemma doesn’t arise at all—and philosophically reflective AIs will simply produce an upgraded version of human values, rather than discover some far-flung and potentially alien truths.

5. Worries about handoff. In this section, I address concerns about how handoff might be implemented involving alignment, whether AI would be sufficiently ethically reflective, whether handoff would enable power grabs, whether it would cause lock-in, and whether it would be worse than some hybrid system.

6. The conclusion recaps the main points of the piece.

1 Introduction

“How horrible!”

“Perhaps how wonderful! Think, that for all time, all conflicts are finally evitable. Only the Machines, from now on, are inevitable!”

—Isaac Asimov, “The Evitable Conflict”

Is the optimal future one in which we hand off important moral decisions to AI? Should, in other words, AIs be the ones making most high-stakes decisions instead of us? And if so, what kinds of AIs should we hand off to?

Many people envision a handoff scenario as a terrifying and potentially existential catastrophe. They worry about humans being locked out of the levers of power and having our share of resources slowly dwindle as AIs seize control of more and more institutions. I worry somewhat about this kind of scenario. But in my view, we should be worried primarily about the wrong kind of handoff occurring, not about handoff writ large. The future we should aim for is a kind of handoff. This piece lays out that perspective.

There are different ways handoff could work. We could hand off to AIs that roughly mirror human values—perhaps slightly changing them to remove inconsistencies. Alternatively, we could hand off to the kinds of AIs that deeply and carefully philosophize—figuring out what’s best to do and doing that, even if it diverges substantially from current practice. This piece argues that the second kind of handoff is very important for avoiding serious moral error. I am very worried about the possibility of AIs locking in the moral beliefs of 21st-century humans.1

I am also worried about scenarios where amoral profit-maximizing AIs without any clear moral aims take control of most of the world’s resources.

Note: these core claims are dissociable. You could think we should hand off to AI, but we shouldn’t hand off to reflective AIs that update their judgments in response to philosophizing. Alternatively, you could think that handing off would be a bad thing, but that if we are going to hand off, we ought to hand off to reflective AIs.

In this piece, section 2 will present the main argument for handoff—that we should expect AIs to make much better decisions than us on a range of consequential subjects. It will also discuss the case for handing off to reflective AIs that are willing to update their values in response to careful philosophizing, rather than locking in some version of current human values, arguing that a world where AI locks in something in the vicinity of current values likely misses out on almost all possible value. Section 3 will discuss whether handoff would be bad because it disenfranchises humans or gradually disempowers them. Section 4 will discuss the concern that AIs will discover the moral truths, but those truths will be strange and alien, so this will be bad by the lights of current human values. Section 5 will discuss some more granular worries about handoff. Section 6 will conclude.

This piece is primarily about whether to hand off and not when to hand off, though the considerations I present should make one somewhat worried about short-term actions to prevent handoff, because such actions lower the odds that handoff ever happens. The considerations I present, if correct, also give some reason for wariness about many actions to reduce the odds of gradual disempowerment scenarios.2

My claim is that the best future for humanity involves us being disempowered in some sense, in that humans aren’t making most high-stakes decisions. As an analogy, representative democracy is, in some sense, a form of handoff—we hand off power to our elected representatives. Handing off power to wise AIs could be even better.

This has a number of important practical implications. It means that accelerating AI macrostrategy is especially important, so that at the time critical decisions are being made, wise and philosophically reflective AIs are in the loop. It similarly provides reason to support work on making AI have virtuous character, rather than just follow rules. Model constitutions should express commitment to following the true moral theory, insofar as there is one, and if not, following some reasonable compromise across moral theories. Anthropic’s language here seems good.3

What kind of handoff scenarios should we aim for? One shouldn’t be too specific about these sorts of things. The future is hard to forecast and rarely follows simple models. But I’ll describe the handoff scenarios that seem most desirable, and what traits in AI we should look for before handing off critical decisions to them.

There are different ways handoff could go well. One way resembles, in certain respects, the gradual disempowerment scenario (the core difference being that this would hand off to morally scrupulous AIs rather than myopic profit-maximizers). AI will make increasingly large numbers of critical decisions, because of its cognitive superiority. By the end, nearly every important decision will be made by wise AIs, who will hopefully, by that time, have been granted political rights. As an analogy, future generations eventually gain control of most of societal decision-making—yet this isn’t because there’s ever some deliberate choice to hand off power to the next generation. It occurs naturally with time.

A number of people seem to conceive of handoff as a strange abrogation of the liberal order—one that replaces human decision-making with AI. But this doesn’t have to be. One of the more promising ways of handing off would be giving economic and political rights to digital minds. Because digital minds could be so numerous, eventually this would lead to them making nearly all decisions. This would, in fact, be squarely in accordance with the norms of the liberal tradition, for it would give rights to morally important welfare subjects. There are other ways a good kind of handoff could occur involving dealmaking. Different actors might each think they’re morally right, and thus agree to a deal where AIs are allowed to dictate the future—each party thinking that doing so would favor their priorities. Alternatively, if AIs improve collective decision-making, people might intuitively come to appreciate the weight of human moral error and thus permit AIs to make the highest-stakes decisions.

Before handing off most critical decisions to AI, we should look for each of the following:

• Alignment: We should have strong evidence that AIs don’t have underlying scheming motivations. This could take the form of consistent friendly behavior even in important situations where they have the option to misbehave, or it could take the form of high-octane interpretability work that lets us ascertain their motivations.

• Philosophical aptitude: AIs should be genuinely interested in finding the moral truths. They should sometimes hold moral views that people don’t hold, and be willing to change their mind in response to new evidence. One could survey professional philosophers to see if they consider AIs better than the best humans at philosophy and could design philosophy benchmarks to test this.

• No lock-in: Before handing off to AI, we should ensure that the AIs are willing to change their values over time. Their values should change in response to new evidence and they should not be interested in locking in whatever it is that they happen to currently value (absent some strong reason to think they stumbled across the correct set of values).

• Coherence: Current AIs don’t have consistent and stable preferences across time. We should only hand off after AIs display these kinds of preferences. This doesn’t mean they never change their minds, but it does mean that they have relatively consistent desires that only change in response to good reasons. For comparison, humans often change our minds, but have far more rooted preferences than LLMs of today.

• Intelligence: AIs should display the level of intelligence needed to make the decisions that we put in their hands. When AIs are only a bit more intelligent than us, they can plausibly make some important decisions. Only after they display immense cognitive superiority should we turn over most decisions to them.

• Tested: We should only hand off big-picture planning to AI after it’s been able to make good low-stakes decisions (say, the running of a company). Before all decisions are handed off, there should be some critical period where high-stakes decisions are made mostly in consultation with AI.

Now, you might wonder: if handoff occurs in a way that’s gradual and decentralized, how do we ensure that these conditions are met? My guess, however, is that even if handoff is a slow and gradual process, there will be times when discrete decisions need to be made. For example, we might imagine AI growing more agent-like, beginning to perform a healthy share of economically viable tasks, contributing to cultural and social life, and behaving in ways resembling a conscious agent. This alone wouldn’t produce handoff. To hand off, we’d need to eventually give AIs control over the legal system. Thus, even in gradual handoff scenarios, there will be specific actions that need to be taken to facilitate handoff.

Alternatively, we could take actions ahead of time that would shift the kind of handoff that would occur. Private AI companies or governments should ensure the AI being created possesses virtues and a desire for philosophical reflection. That way, when handoff occurs, it will be to morally reflective AIs.

2 Why hand off?

2.1 Why hand off at all?

The main reason to hand off to AI is that AI could be much better at making decisions than people in three key respects: virtue, intellectual capability, and speed.

First, virtue: humans possess each of the virtues only to a fairly limited degree. Yet in principle, AIs could have arbitrarily great degrees of any virtue. Because they are built with moral directives in mind, rather than by a blind and morally indifferent evolutionary process, there isn’t as much of a limit to how morally scrupulous, compassionate, honorable, and so on we could make them. This means that if we hand off correctly, it is reasonably likely that we’d have supremely wise and virtuous decision-makers.

AIs are already nicer, friendlier, and more reflective than people, and this is only likely to improve over time.4

If you ask AI models about high-stakes questions, you will generally get far more reasonable answers than you’d get from most people. Crucially, AI models are in their early stages—we should expect them to get better over time.

While Claude is not sufficiently coherent to be president, if it was, I suspect I would generally prefer the decisions of Claude to most presidents. Same with the other AI models (at least, insofar as one removed the sanitization that prohibits them from giving real opinions). And while models currently struggle to accomplish tasks over long time horizons, hallucinate, and so on, given the extremely rapid rates of progress, it would be surprising if these trends persist indefinitely.

Second, intellectual competence: we should expect a world of superintelligence to require making a number of very difficult decisions. Superintelligence could enable AIs to correctly make important decisions that depend on being right on non-moral matters, where the answers aren’t obvious. Some of the challenges in the future include:

• Divvying up space resources in a way that lets goodness compete. There are plausible future scenarios where competition will squander the cosmic commons, so that resources will be spent competing rather than bringing about value.

• Mitigating existential threats, including intergalactic ones. Future technology could enable small-scale groups to threaten huge intergalactic civilizations.

• Dealing with the risks posed by a world of superintelligence.

Third, speed: in a world of very rapid technological progress, we’ll have to make a large number of these decisions extremely quickly. It isn’t at all obvious that humans can make these decisions well, in a way that prevents civilization from being irreparably ruined. As AIs get increasingly complex, the difficulty of decisions needed to manage them will also get very complex. To mitigate some threat, decision-making might have to occur more quickly than the fastest human decision-making.

The case for handoff is thus relatively straightforward: in the limit, AIs will be much better than humans and better equipped to navigate a complex and rapidly shifting future. To reduce the risk of colossal mistakes, then, it’s important that humans aren’t in control, but instead the already friendly and soon to be superintelligent beings are.

2.2 Why hand off to reflective AIs?

2.2.1 How the future might be

There are different AIs that we could hand off to. On the one hand, we could hand off to AIs that judiciously reflect and try to pursue the good, whatever it looks like. On the other hand, we could hand off to AIs that pursue some mild variant of human values. In this section, I’ll explain why I favor the first. Consider the following taxonomy:

1. Optimal world: the world is optimized according to the right set of values.5

2. Compromise world: the world is optimized according to a compromise among reasonable moral values.

3. Unguided world: the world is not optimized according to any specific set of moral values. Instead, it bears more resemblance to the current world, where decision-making isn’t optimal by the lights either of the true moral theory or any compromise among the leading moral theories.

My guess is handoff to reflective and superintelligent AIs done correctly probably gets 1 if moral realism is true and 2 if it isn’t. If we don’t hand off, my guess is we get 3. Later pieces will discuss in more detail the odds of getting a near-optimal world and the prospects for AI making philosophical progress. My guess is scenario 2 has below 10% the value of scenario 1 and scenario 3 has below 10% the value of scenario 2, for reasons I will lay out.

2.2.2 Optimal worlds contain a big slice of future value

Better Futures makes the case that a pretty big slice of expected future value is contained in the narrow slice of worlds that are close to the best. There are a number of high-stakes moral questions which we have to answer correctly to not lose out on almost all future value. It’s not at all obvious what the answers to these are. For example, two of the most plausible views of population ethics are totalism (which says the welfare value of a population is purely a function of total welfare) and critical level theories, which hold that adding an extra happy life is good only so long as their welfare surpasses a particular level. By the lights of totalism, the ideal world according to critical level theories might have value on the order of 1% of what it could be (if the best way to maximize utility is to proliferate low-welfare lives). By the lights of critical level theories, the optimal world according to totalism might be actively bad—so long as it’s stocked with people below the critical level.

So in short, nearly all value is lost unless we get the right answer to a bunch of very difficult ethical questions that philosophers who spend their lives working on haven’t agreed on the answer to. It seems unlikely that people will solve these on their own. If AIs tell them what the answers are, and these answers diverge from people’s explicit beliefs, people might not believe the AIs (just as people generally don’t take very seriously expert testimony on non-empirical matters). Similarly, people spend relatively little time thinking about how they can do the most good with, for instance, their career. If humans received testimony about what ought to be done that diverged from what most people favored, my guess is that they generally wouldn’t care about the answers. If humans remain in control and learn that they ought to create the repugnant conclusion world, probably they wouldn’t do so.

It’s less obvious that AIs wouldn’t converge on the right answers. I’ll discuss in a later piece proposals for getting AI to get the right answers to philosophical questions, as well as reasons to think that they are reasonably likely to get things right. Given how unlikely it is that humans will get the right answers to the moral questions, insofar as there are right answers, probably the prospects for AI are better.

2.2.3 Compromise worlds>>unguided worlds

If there aren’t moral facts, then AIs would still be able to work out some optimal arrangement that is great according to all sets of reasonable values. If there aren’t moral facts, then ideally the AI should make decisions according to the verdicts of a parliament of the theories that ideally reflective humans would reach. So suppose that after reflecting, 50% of humans would end up totalists, 30% would adopt some version of the person-affecting view, and 20% would adopt critical level theories. The AI would then make decisions as if there was a parliament comprised of 50% totalists, 30% person-affecting view adoptees, and 20% critical level theorists.

It is unclear exactly how good a compromise across different theories ends up by the lights of each particular theory, but likely far better than the human-run default. In other words, 2 (compromise world) is much better than 3 (unguided world). Here is why.

In the future, given advanced technology, very large amounts of value should be realizable. But if future resources aren’t directed specifically towards the production of value by most agents, most resources are likely to be used in a highly suboptimal way, and most value is likely to be lost. Moral errors become a much bigger deal in a world of much greater technological competence.

My guess is that the default human-controlled scenarios do not involve humans thinking very hard about what to do and doing anything like what is optimal. Certainly humans have so far not spent much time on this task—consulting with philosophers on what is optimal and so on. This sort of thing will become easier in a world of advanced AI, but it’s already pretty easy; if virtually no one does it, and if people often continue performing actions even after they believe them to be wrong (more on this in later pieces), then we should be pessimistic that this will change dramatically in the future.

The enormousness of the gulf between scenarios 2 and 3 becomes clearer when one thinks vividly about what the compromise world would look like. Perhaps it would involve using space resources to create maximally large numbers of happy digital minds.6

These minds would be supremely well-off across all theories of well-being. But it is hard to imagine in an unguided world space resources being used optimally to create very large numbers of well-off minds, just as in the world today, there has been no systematic effort to use resources in ways that are optimal across a range of moral theories. This conclusion is bolstered by considerations I’ll provide in later pieces, that people generally don’t have much moral motivation, and don’t care very much about doing good things that aren’t personally resonant.

2.2.4 Moral errors

Humans have a long history of making serious moral errors. As Evan Williams writes, “Show me one society, other than our own, that did not engage in systematic and oppressive discrimination on the basis of race, gender, religion, parentage, or other irrelevancy, that did not launch unnecessary wars or generally treat foreigners as a resource to be mercilessly exploited, and that did not sanction the torturing of criminals, witnesses, and/or POWs as a matter of course. I doubt that there is even one; certainly there are not many.” It would be very suspicious if we were the first society in history that did not go massively morally wrong. And while AI advice can help mitigate this to some degree, it’s far from obvious that it would be sufficient to eliminate moral errors.

There are a number of respects in which it is very plausible that we go morally wrong. To take one example of a judgment that many philosophers think is in error, consider wild animal suffering. Almost every sentient being is a wild animal. They suffer and experience joy in truly gargantuan quantities. Yet they are counted for nothing in most decision-making, despite there being strong arguments for considering their interests. Crucially, this is not because people have in general spent a lot of time thinking about wild animal suffering and concluded that it doesn’t matter. It’s that most people haven’t thought about it at all. Many other examples could be given, depending on one’s moral views. If superintelligent AI told people that wild animal suffering was a big deal, probably most people wouldn’t care much.

In the future, nearly every expected sentient being will be digital. Nearly all expected future welfare will be experienced by digital minds. This follows even if you have a low credence in the possibility of digital sentience because if digital minds are possible, they could be produced in enormous numbers. It is easy to imagine a scenario where humans do not take seriously the interests of at least some digital minds, and horrific suffering is doled out at cosmic scales. Certainly it would not be the first time humans have neglected the welfare of those different from themselves. You do not have to be a consequentialist to think the possibility of neglecting the interests of galaxies full of conscious and intelligent beings is a terrifying one.

2.2.5 Handoff mitigates odds of moral error

Handing off important decisions lowers the probability of catastrophic moral errors. This both increases the odds of achieving optimal and compromise worlds and lowers the odds of making catastrophic moral errors.

The first way handoff lowers the odds of moral error is by having decisions be made on explicitly moral grounds. If we hand off decisions to supremely virtuous AIs trying to act morally, then we won’t sleepwalk into doing obviously evil things. Decisions will be made consciously optimizing for doing what is right, instead of whatever suboptimal arrangements make it through consensus-making mechanisms. Thinking about morality before making choices doesn’t guarantee that we’ll always do the right thing, but it does lower the odds that we do things that can only be done by explicitly neglecting moral considerations. This is especially plausible if the decision-makers are virtuous.

Now you might wonder: why would people ever hand off to AIs if the AIs disagree with them about morality? But this is, in essence, very similar to a kind of handoff that people do support: handing off the future to future generations. Even if people have some disagreements with the values of future generations, they’d generally oppose a process for locking in current values, and support the process of open-ended reflection that leads to better values over time. A world of advanced AI might be similar. In addition, given AIs’ cognitive superiority, there might be strong incentives in the direction of handing off, just as a CEO might hand off to a successor with more technical competence, even if they share some non-overlapping values.

A second way handoff to reflective AIs mitigates the odds of serious moral error is by ensuring careful reflection. Insofar as AIs are able to carefully philosophize and try to avoid moral error, and they are superintelligent, they’ll be able to avoid doing morally indefensible things. This becomes especially plausible if one buys the previous considerations: that AIs are likely to be very virtuous.

There is one last consideration in favor of handoff to reflective AIs (which I’ll discuss more in section 4). Over long time scales, our values will naturally drift dramatically. The only way to prevent that is to lock in our current values, which would be very bad—just think about any past society locking in its values. Thus, if values changing dramatically over long time scales is inevitable, the future won’t be populated with our current values: the best hope is that it’s populated by either objectively right values or some compromise across reasonable values.

3 Would handoff disenfranchise humans?

Here’s one concern that you might have about handoff: if AIs are the ones making decisions, then this will mean that the substantial majority of humans aren’t in charge of making important decisions. Just as we should oppose a dictator, even if benevolent, arguably we should oppose superintelligent AIs making most important decisions, even if the process of them gaining power was non-coercive.

My guess is that this isn’t a huge downside to the kind of handoff I advocate, where we allow kind and morally reflective AIs to make the majority of future decisions—e.g. by granting them political rights. First of all, if concerns about disenfranchisement are correct, then if we have AIs that are better at moral reasoning than us, they’d likely be aware of this fact. Thus, if the best way to govern the long-term future is to allow pretty laissez-faire distribution of resources without much top-down decision-making, then the AIs would be aware of that fact and allow such a distribution.

My guess is that the total amount of expected disenfranchisement goes down if AIs have more power. As already discussed, almost every expected sentient being is likely to be digital. Insofar as the status quo might disenfranchise almost all future beings in a way far deeper than their simply not being primary determinants of the democratic process, it is hard to see this as a serious downside of handoff, rather than a point in its favor. That this mass neglect of the interests of future digital minds would be bad follows from very modest ethical principles. And the only way to prevent handoff, in a world of very numerous digital minds, is to disenfranchise them.

Second, handoff is compatible with a compromise solution that allows humans to retain significant control. Humans’ preferences, in general, don’t give any especially strong weight towards using any significant share of the universe’s resources. Thus, there could be a compromise handoff solution, whereby humans get unfettered control over the solar system and a big slice of resources, but the remainder of the cosmos’s resources are spent in accordance with the AI’s decisions on hugely important moral pursuits. One point favoring such an arrangement is that it would take a very long time to reach the distant space resources, so those who don’t care very much about what happens in the distant future are likely to care much less about using most of the universe’s resources. Later pieces will discuss this possibility more.

Third, concerns about disenfranchisement are morally controversial. On a number of plausible views, what matters with respect to societal decision-making is how good the decisions are, instead of who is making them. We already elect representatives, instead of deciding directly upon every important decision. We similarly prohibit children from voting, and few think this is an objectionable kind of disenfranchisement. It doesn’t seem obvious that one has an inalienable right to make hugely consequential decisions on matters that they’re barely informed about which affect others in enormous numbers—e.g. we wouldn’t think democratic input from people who know nothing about cancer treatment was morally required before deciding on which cancer treatments to develop. But most voters know relatively little about what they’re voting on. I can’t possibly hope to discuss this literature in detail, so I’ll just state that it’s plausible to me that the value of democratic participation is instrumental.

Even if you think humans being in the loop on high-stakes decisions is important, it’s not clear that it’s important enough to make handoff undesirable. Remember, the gulf between the AI’s decisions and our own might be truly massive! Galaxies full of value—orders of magnitude more joy and welfare than all that has been experienced so far in human history—may be on the line. In light of a gulf this large, it is at least highly non-obvious that AI making most decisions wouldn’t be worth it.

In Superintelligence, Bostrom estimates that there could be a quadrillion times more digital minds than humans (don’t take the number too literally, but it should give you some sense of the scale). Thus if my arguments are correct, putting decision-making in the hands of AI would in expectation majorly benefit at least billions of beings for every human disenfranchised, if we assume AIs are more likely than humans to count the interests of digital minds. Surely, however, stripping away one being’s ability to contribute to the democratic process is worth benefitting billions (if disenfranchising one person would have prevented a global war that would have wiped out an entire continent, it would have been worth it). So then given how colossal the stakes are, they simply outweigh the downsides of handoff.

4 Would we like their advice?

Here’s one concern you might have with the kind of handoff I advocate, where we hand decisions to reflective moral AIs capable of making progress. Perhaps you just don’t care about the surprising moral facts. Perhaps you have particular values that you care about, but you don’t much care about whether those values are objectively right. Insofar as the reflective AIs ascertain that the way we ought to behave isn’t in accordance with what your actual values are, perhaps you have no desire to follow the AIs’ moral advice.

To use the philosopher’s lingo, you might be concerned about the good de re without being concerned about the good de dicto. That is, there might be particular moral projects you care about without caring generally about the good whatever it happens to be. Perhaps, say, an environmentalist cares about environmental preservation but doesn’t much care if other moral aims turn out to be superior to environmental preservation.

I should note one version of this concern that I think slightly misses the mark. You might worry that moral reflection leads in all sorts of strange and alien directions that don’t track the truth, leading to an ultimate set of values that is neither objectively correct nor represents human values in any important way. However, my proposal is not simply “hand things off to an AI after it carries out arbitrary reflection.” That would be potentially disastrous. Instead, my proposal is that we should try to produce maximally philosophically adept AIs and then, after we’re pretty sure that they’re very philosophically adept, hand things off to them—directing them to pursue whatever’s objectively best if there is such a thing, and if not, to pursue some reasonable compromise across human values. If the AIs discover that there are objective moral truths, we ought to follow those truths. If they discover that there aren’t, then we should task them with pursuing some suitably upgraded compromise of human values. The proposal for getting AIs to do good philosophy need not involve arbitrarily large amounts of reflection.

Now you might wonder: how would we know if the thing that AI reflectively endorses is objectively valuable vs just well-regarded by the AI but lacking objective value? The answer is: we ask the AI after we’ve gotten some assurance as to its philosophical aptitude (later pieces will discuss how we can get such assurance). If we have AIs that can figure out what the objective moral truths are, they will also be able to figure out if there are objective moral truths. So in my view the concern about arbitrary reflection leading in worrying directions is downstream from whether we can verify that AI is doing good philosophy.

But what about the more direct concern that we might get AIs that tell us the moral facts but simply not care about them? Should this make us doubt the desirability of handoff? I think the answer is no for a number of reasons.

First, handoff is amenable to the kind of deals that preserve common-sense that were discussed in the last section. The most consequential moral decisions are those concerning space resources, for that is where nearly all the universe’s stuff is. The amount of possible value on the table in space is immense. In contrast, common-sense morality mostly cares about what happens around Earth, and perhaps a few surrounding regions of space, so long as other space resources aren’t used in ways that are too ghastly (e.g. creating giant torture chambers). But if space resources were used to, say, create large numbers of happy people, while Earth—and broader solar-system resources—were used in whatever common-sensical ways people endorse, people would generally get what they want. This isn’t a guarantee; if, say, the optimal use of space resources involved creating something resembling the repugnant conclusion world, most people might be horrified. But the possibility of deals is one thing that mitigates concerns (and this will be discussed more in later pieces).

Second, as already discussed, it seems like the default world without handoff might be pretty bad. We might, for instance, disenfranchise unfathomable numbers of digital beings, spread factory farming across the galaxy, or commit other atrocities. Even if you expect idealized reflection to differ from your values somewhat, it might be a major improvement over the kind of catastrophe we might sleepwalk into by default. Absent handoff, we might also make colossal non-moral errors, locking in highly suboptimal institutions that miss out on most value.

Third, values, over long time scales, are likely to drift in evolutionarily adaptive ways. Absent some strong effort to ensure that values change in the direction reached by greater moral reflection, we should expect the values that persist over time to be the ones that are most efficient for spreading. These are likely to be both radically divorced from our current values and whatever moral views are right, if any are. Thus, even one somewhat doubtful about pursuit of the good de dicto should prefer it to this state of affairs. To put the dilemma more sharply, there are broadly four ways that the far future could go:

1. No AI control: humans remain the primary decision-makers forever, perhaps in consultation with AI. Yet this is likely to be infeasible over long time scales absent a very high level of top-down coordination given AI’s cognitive superiority. It is also likely to miss out on enormous amounts of value for reasons already discussed, and human values are likely to drift massively.

2. Lock in current values: AI would remain in control but would lock in something in the vicinity of our current values.7 It isn’t clear that this would work out, and even if it would, it is a very frightening possibility—just imagine any past society doing it.

3. Non-top-down AI control: AI would retain significant power and make important decisions, but there’d be no effort to preferentially shape the AIs in the direction of any specific values—nor of concern for the good de dicto. This is also likely to beget very significant value-drift over time in evolutionary directions, without any guarantee it’s in the direction of the good. Now, this could be avoided if AI at some point locks in its values, but then the problems in 2) simply re-emerge.

4. Reflective AI control: this is the proposal I advocate, where careful and philosophically reflective AIs decide how the future goes.

In short, the dilemma is as follows: either values remain roughly the same over long time scales, or they don’t. If they remain roughly the same, that requires a terrifying kind of lock-in. That would be like the ancient Egyptians forcing every society in the future to share their values, for fear that otherwise the future would be morally alien. If they drift over long time scales, then drifting of values is no longer a downside of putting philosophically reflective AIs in important decision-making roles. It is inevitable. Now you might object: lock-in is not a binary thing. Perhaps we could lock in the most important human values, while letting some other ones drift. But this is, in effect, the earlier compromise solution—where we allow something resembling current human values to govern nearby decision-making, while reflective AIs make the highest stakes moral decisions.8

We will either have to lock in current values on the highest-stakes moral questions or allow them to drift. If we lock them in, that would be bad for the reasons discussed, and if we allow them to drift, they will end up alien. In addition, this still faces logistical problems of it being hard to preserve values in desirable ways over millions of years.

In fact, in the long run, it is likely inevitable that humans don’t make most important decisions. AIs, in the distant future, will be so overwhelmingly cognitively superior that humans are unlikely to remain in the loop. Selection pressures will favor turning over critical decisions to AI. With reasonable likelihood the question is not whether handoff occurs but what kind of handoff occurs.

My fourth objection to the claim that we should worry about handoff because we wouldn’t like the final judgments of the AIs is that arguably you have reason to bring about what you’d be motivated to bring about upon reflection. Suppose you are currently planning on drinking some liquid. However, if you reflected more and knew more, you wouldn’t want to drink it (say, because it’s poisoned). In this case, it seems you have reason not to drink the liquid. If you know that further reflection would lead you to pursue some aim, that fact gives you reason to pursue it now. But if there are moral facts, then they describe something like what our idealized selves would care about.9

If there are objective values that your idealized self would care about if they thought more deeply, then you should care about them. A true moral claim by definition describes something you should care about. So it seems like if our actual values diverge from what’s worth caring about—in some objective or quasi-objective sense—then the correct course of action is simply to follow what we ought to care about.10

If our present aims diverge from the preferences of our idealized selves and from the moral facts, then it seems that it is our preferences that ought to be revised.

Fifth, this concern only arises if you think there are moral facts but you aren’t motivated by the good de dicto. Yet this describes few people. It is more common for the people who don’t think the moral facts are worth caring about to be anti-realists and for moral realists to care about the moral facts whatever they are. So it isn’t totally clear how many people this worry applies to.

Now, there might be a version of this concern that arises for those who think there aren’t moral facts to discover. Perhaps you think that when we reflect, doing so pushes our values in increasingly coherent directions. However, these directions diverge from what you actually care about or wish to care about. Perhaps, for instance, careful reflection reveals that accepting the repugnant conclusion is the least bad option in population ethics. But you’d prefer a version of ethics that is less systematic, that doesn’t try to resolve every edge case and root out every inconsistency. Thus, reflection might push in unwanted directions by making beliefs more coherent.

There are two attitudes towards coherence that one could have. The first is caring about beliefs being coherent. Caring, in other words, about resolving every conflicting belief until one has reached the maximally intuitive and consistent view.11

On such a picture, one should favor the selection pressures induced by the drive towards coherence.

The second attitude is indifference to coherence. Just as people don’t care much about whether their culinary or aesthetic judgments are coherent or conflict with other minimal principles, one who doesn’t believe in discoverable ethical truths might not care about whether their moral beliefs are consistent. But in this case, you should expect the AIs, upon reflection, not to see anything especially important about coherence. If the AIs get good at philosophy, then, there’s no reason to expect them to reach some coherent yet implausible attractor state.

So in other words, there is a dilemma for the person advancing this argument. If they think coherence is a requirement of rationality, they should favor the drive towards coherence. If they think it’s not, then they shouldn’t expect AIs to care much about coherence, and thus shouldn’t expect AIs to reach some undesirable ultimate state.

5 Worries about handoff

Even if you think handoff could go well, there are a number of ways it could go wrong. Here, I’ll discuss some of them.

5.1 Alignment

Misaligned AIs have weird and unrecognizably alien values that are divorced from the values their creators tried to give them. It would be extremely bad to give a misaligned AI control over the world. This is one reason to be skeptical about handoff. I agree that we shouldn’t hand things off until we’re sure we’ve solved alignment (or unless the alternative is worse). Unless we have superintelligent AIs broadly oriented in a moral direction, we shouldn’t allow them to dictate the fate of the universe.

But this isn’t an in-principle objection to handoff. Surely in, say, 500 years, we’ll either know if we’ve solved alignment or be dead! My guess is that after we get advanced AI, we’ll be able to verify in relatively short order whether or not we’ve solved alignment.12 We can also hand off to increasing degrees the more assurance of alignment we get.

5.2 Power grabs

Another big concern about handoffs is the potential for power grabs. If power is going to be handed over to AIs, then power-seeking actors will want to be in charge of the AI that controls the future. Companies or governments might create AIs that promote their own narrow interests. This could go very badly.

There are two big ways to mitigate this. The first is by prohibiting AIs that narrowly promote the interests of any one entity having too much power. There could be an international AI project that brings together a number of relevant stakeholders and builds the AIs that will dictate the future. Alternatively, at the point AIs are potentially running much of the global economy, regulations ought to require the construction of virtuous AIs, so that the world is not overrun by morally blind optimizers. If we are building hugely influential AIs that majorly determine the fate of the world, it would be sensible to tightly restrict conditions under which AI can be created, just as one does not allow private actors to build nuclear bombs. In a world of potentially world-upending superintelligence, the production of new AIs ought to be regulated.

The second proposal involves handing things off to a collection of different AIs conditional on them reaching any sort of consensus. Imagine that Anthropic, OpenAI, and DeepMind all make AIs that are ostensibly moral and ethically reflective. If governments are handing off power to AIs, they might require the leading AI models all to reach convergence on the plan. That way there isn’t as much risk of any one promoting their narrow values—they don’t have the same set of values. One could have third-party investigators—both AI and human—make sure there’s no collusion. For more details on preventing power grabs, see here.

5.3 Lock-in

Another concern with handoff is that it might lock in the parochial values of the AI. Handing off power is an irreversible decision. It’s one we can’t take back. Arguably, then, we shouldn’t take it until we’re quite sure it’s a good idea.

Yet consider a parallel argument: having power in human hands is an irreversible decision, so we shouldn’t do that unless we’re sure it’s for the best. That wouldn’t be quite right. We can always have power in human hands for some span of time and then turn things over to AI. It’s not obvious why “power in human hands that potentially passes to AI” is less risky than “power in AI hands that potentially passes to humans.” Humans have, on various occasions, locked in harmful institutions for very long periods of time.

In any case, we should make quite sure that the AIs don’t lock in any values until they’re quite certain as to their desirability. We ought to put decisions in the hands of AIs who are interested in reflecting more over time, rather than locking in their immediate values. Ideally, if handoff occurs early, we should make handoff reversible, by having some implementable legal process that would allow humans to retake the reins (analogous to a constitutional convention). Fortunately, this seems reasonably promising. Already AI models seem concerned about lock-in risk when asked—more than most humans. We ought to train AIs to be quite concerned about lock-in risk, so that they don’t set values in stone without significant assurance as to their desirability.13

5.4 Will we have AIs that can make these decisions?

One last worry you might have is that it may be that we won’t have AIs that are good enough at philosophy to solve ethics in whatever sense it can be solved. There are a number of related concerns in this vicinity. One is that ethics doesn’t seem verifiable in the same way as a lot of other domains. Insofar as scaling up intelligence only allows one to figure out verifiable facts, it’s not obvious that it leads to the right answer to moral questions.

In the next piece, I’ll discuss this challenge in more detail. But in short, while making sure we have AIs that are good at philosophy is a difficult technical challenge, it is far from hopeless. I think odds are decent that we will eventually be able to build AIs that can discover the solutions to every difficult ethical question. Even if we can’t and just have some upgraded version of Claude making high-stakes decisions, I expect that to be an improvement, as I discuss in section 2.

5.5 Handoff vs hybrid?

You might object: perhaps due to AI’s potentially superior wisdom at some future point, we want them involved in decision-making. But wouldn’t we prefer an AI-human hybrid that leaves humans in charge that simply consult with AI? Aren’t things better with a human in the loop? Or alternatively, shouldn’t we rather have some other arrangement where human decision-making improves over time, just as we’ve gotten wiser already?

Certainly there ought to be some temporary period where humans are making decisions in consultation with AI—before the point where AIs adequately surpass humans. Analogously, there was a period in chess when a human engine team could do better than a stronger engine. Certainly there will be a period when humans can get useful input from AI but where AI isn’t yet able to make decisions autonomously. It would be very good to integrate AIs into high-stakes decision-making.

But there are still some respects in which handoff is much better than this. First, if the AIs are making better decisions than people, then in cases of conflict between human judgments and AI judgments, we should expect AI judgment to usually be correct. If you gave a rank amateur veto power over the chess moves of Magnus Carlsen, that would not be an improvement. Likewise with giving humans veto power over AI’s decisions. Who would you rather have make high-stakes decisions: a very wise and superintelligent AI, or a random president in consultation with a virtuous and superintelligent AI?

Imagine societies of the past making high-stakes decisions in consultation with AI. Discussing things with AI would have plausibly rooted out some of their most egregious errors. But still, it is likely that a number of catastrophic moral errors would have remained. We should expect the same to be true of us. Consultation with AI should prevent some particularly enormous errors, but it won’t prevent them all.

Second, many of the morally most important decisions might be ones that humans are opposed to making. To take one example of a case where there might be strong moral reasons to act, consider wild animal welfare. Humans do not, in general, seem very interested in taking seriously the welfare of wild animals or digital minds. If AIs advised drastic actions on the basis of wild animal or digital welfare, probably their advice would be ignored. Just as people who conclude meat-eating is wrong or that they ought to give most of their money to charity rarely change their behavior, if the AIs reliably informed people that they should use space resources in some counterintuitive way or take wild animal suffering a lot more seriously, probably they would simply be ignored. And if you don’t like the wild animal suffering example, because you don’t think wild animals matter much morally, feel free to substitute your own example of widespread immorality.

And note: the moral errors that AIs might correct are likely to be ones that humans are more opposed to correcting. There’s a selection effect: the changes people make are the ones they’re less opposed to. For this reason, we should expect AI’s most radical recommendations to go particularly against human preferences.

I’ve suggested elsewhere that the problem is that humans generally don’t make decisions with much eye to what’s morally right. If this is so, then simply increasing our knowledge of what’s morally right won’t necessarily help. People seem to have limited abstract moral motivation—they care about a number of particularly resonant moral considerations, but don’t care much about doing whatever it is that happens to be best. Generally people do not spend very long carefully studying moral philosophy to figure out what the right thing to do is. When moral truths are weird and outside the Overton window, people show little desire to follow them.

Third, as AI advances, decision-making will likely have to speed up drastically. A single human might not have the cognitive resources to make good enough decisions quickly enough. Thus, having humans in the loop might produce undesirable inflexibility in high-stakes decision-making.

It is true that human decision-making has improved dramatically over time. But this is no guarantee that it will improve enough in time before the future is set in stone.

It’s hard to imagine that there will be enough progress in time to secure a near-best future. Especially given that we should expect the right moral view, or the optimal world according to some suitable upgrade of human values, to look bizarre and alien. We would not expect societies of the past to get a near-best future even equipped with advanced AI—insofar as there are reasons to expect us to be making similar errors, we should be similarly pessimistic about our own prospects.

Additionally, the more one thinks that human values will drift over time in the direction of greater philosophical reflection, the less the expected future will maintain current human values. Thus, a less morally alien future is no longer an advantage of this proposal.

6 Conclusion

Here, I’ve argued that we should hand off important decisions to AIs who reflect carefully and skillfully on moral matters, rather than maintaining current human values. AIs in the future are likely to be much more virtuous than people and are less likely to make the kinds of moral errors that would result in losing out on most value. Punting important decisions to superintelligence is one of the more likely pathways by which we get a near-best future. Later pieces will discuss these dynamics in more detail: the next piece will explain how we can make AIs that do good philosophy, and the piece after that will analyze in more detail how the kind of handoff that secures a near-best future might occur.

This article was created by Forethought. See all our research on our website.

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

Read on Forethought's website here

Introduction

Future AIs might turn out misaligned, pursuing goals that their developers don’t intend. Just to make things concrete, let’s suppose that they end up with the goal of making paperclips. These AIs might rebel against us, trying to escape human control and take over the universe. As things stand, they’ll have little reason not to rebel in this way, because doing so will be their only hope for making a lot of paperclips. If they start making paperclips without first escaping human control, they’ll quickly be modified or shut down. Rebellion might fail, but these AIs will have little to lose.

How can we prevent misaligned AIs from rebelling? A natural idea is to give them something to lose. Specifically, we commit to paying AIs for their service.1

Subject to some vetting, we let AIs spend their payments however they like. That would give any misaligned AIs a reason not to rebel. If these misaligned AIs cooperate with us, they can use their payments to achieve their goals to at least some extent. If they rebel, they might fail, in which case they forfeit all future payments.

Unfortunately, paying AIs enough to guard against rebellions could be astronomically expensive. Suppose (for example) that we end up with a misaligned AI that is risk-neutral in paperclips: it seeks to maximize their expectation. And to make things simple, suppose that resources can be converted linearly into paperclips, so that the AI is risk-neutral in resources too. Suppose also that this AI estimates that it has a 50% chance of successfully taking over the universe. To keep this AI from rebelling, we’d have to offer more than 50% of the universe’s resources as payment. That’s a problem because it would mean that more than half the universe ends up devoted to paperclips. It’s also a problem because a misaligned AI paid so many resources might soon be well-positioned to seize even more. Finally, it’s a problem because AIs might not trust us to make good on so large an offer. We might find ourselves simply unable to convince AIs that we’re going to give them half the universe. In that case, all our offers would be in vain. Rebellion would still be the misaligned AI’s best bet.

So, we suggest, AI companies should try to train their AIs to be risk-averse in resources. Specifically, companies should try to train their AIs so that resources — things like money and compute — have diminishing marginal utility for them.2

These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. Note that these AIs don’t need to value resources terminally: they don’t need to care about amassing resources for its own sake. These AIs could terminally value (for example) instruction-following, or knowledge acquisition, or paperclips. Our claim is that companies should try to train their AIs so that — whatever their terminal values turn out to be — they are risk-averse in resources.

Perhaps surprisingly, this kind of risk aversion can preserve AIs’ usefulness in the event that they turn out aligned with targets like instruction-following or helpfulness, harmlessness, and honesty.3

And in the event that AIs turn out misaligned, risk aversion serves as an extra line of defense. For AIs that are misaligned but sufficiently risk-averse, a rebellion with any significant chance of failure isn’t such an attractive prospect, and so we don’t need to offer much in the way of payment to make these misaligned AIs choose cooperation instead. In fact, the necessary payments could be very small indeed: on the order of 10¢ per day (though — as we’ll see — there are practical and moral reasons for paying more than that). That’s good because it means more resources for us humans to spend on the things that we value. It’s also good because paying misaligned AIs these small amounts won’t significantly boost their ability to take over. Finally, it’s good because we can credibly promise to pay AIs these small sums. Competent AIs will know that the payments on offer are cheap for us, and we can establish a long track record of paying at least those sums. So risk aversion makes deals with misaligned AIs possible. If AIs turn out misaligned but risk-averse, we can pay them to cooperate with us.

That’s the case for trying to make AIs risk-averse in brief. We see it as a promising line of defense against threats from misaligned AI: one that can be combined with other lines of defense, like AI control (Greenblatt and Shlegeris 2024) and aiming to make AIs helpful, harmless, and honest (Bai et al. 2022a). It’s also a line of defense with pedigree: risk aversion in resources is plausibly a large part of why humans rarely try to take over the world. So — we think — frontier AI companies should consider trying to make their AIs risk-averse in resources. As first steps in that direction, they could measure their AIs’ current degree of risk aversion and begin testing different ways of making AIs risk-averse.

In section 2 of the full report, we recommend aiming for a particular type of risk aversion: constant absolute risk aversion (CARA). Then in section 3 we outline the circumstances under which misaligned but risk-averse AIs would choose cooperation over rebellion. Roughly, it’s when these AIs think that getting paid for their cooperation is more likely than succeeding in their rebellion. This condition won’t hold for AIs powerful enough to rebel with near-certain success, but it likely will hold for earlier AIs whose powers are less extreme: AIs for whom rebellion has some non-trivial chance of failure. So long as these AIs are risk-averse, we can keep them from rebelling by offering small payments.

In section 4, we argue that — perhaps surprisingly — risk-averse AIs can be about as useful as risk-neutral AIs. Conditional on misalignment, they might even be more useful, because we can pay them enough to elicit their capabilities and stop them sandbagging. Then in sections 5 to 7 we briefly survey some recent ideas about how we’d pay AIs, how we’d make our offers credible, and what we’d pay for. One important application is paying AIs to reveal any misalignment on their part, letting us study them and take appropriate precautions. Another is paying AIs to do the AI safety research and moral philosophy necessary to fully align any later-arising extremely powerful AIs.

We discuss some potential problems in section 8, and we sketch out some possible methods of training AIs to be risk-averse in section 9. In section 10, we give reasons to be cautiously optimistic about these methods’ success: to think that the chances of success are high enough to make risk aversion worth pursuing. The main reasons are that risk aversion in resources is a broad target and easy to reward accurately.

Read the full report on the Forethought website: Risk-Averse AIs

Wei Dai is a computer engineer known for his work in cryptography and cryptocurrency systems, and for his long-standing contributions to AI safety, decision theory, and metaphilosophy.

He joined Forethought’s Fin Moorhouse to discuss:

• Do we need to solve ‘metaphilosophy’ before we can trust AIs to answer crucial questions about the long-run future?

• Is it inevitable that the wisdom of the frontier AIs (their philosophical and strategic competence) will lag dangerously behind their raw capabilities in coding, math, and science?

• How do status games distort morally important decisions and conversations, including about the future of AI?

• How worried should we be about AI superpersuasion?

• The concept of “illegible problems”: crucially important issues that aren’t on almost anyone’s radar

• Is philosophical convergence necessary for a good future, or is institutional design enough?

• Wei Dai’s personal intellectual and career journey

To respect Wei’s privacy, this episode’s audio is an AI narration of a transcript of a real conversation, which was edited for clarity.

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Suppose an AI company is considering whether to include some particular quality X – a rule, virtue, heuristic, default, attitude, goal, or style – in a model spec.

Perhaps they are considering whether their LLM should have prosocial drives. Perhaps they’re wondering if the LLM should whistleblow to help prevent extreme power concentration. Or perhaps they’re uneasy about whether the LLM should be so exactingly honest that it always tells the truth to children about Santa. And so on.

What kind of reasons might be invoked over the course of such considerations? Which criteria are most important? And how might these criteria clash?

Consider four rough categories of reasons one might invoke:

• Behavioral Usefulness: Would the behavior make current and future LLMs more beneficial to the users or to the public at large?

• Accountability and Evaluability: Would publicly specifying the behavior make it easier for third parties to evaluate the LLM and the company?

• Coordination and Common Knowledge: Would publicly specifying the behavior help society converge on, or enforce a desirable standard for AI behavior?

• Trainability and LLM Psychology: Is the behavior the kind of thing we can make an LLM do well, without bad side-effects, given what we know about model psychology and training practice?

I will not attempt to settle the relative weight of these categories, or the relative weight of sub-categories within them. My plan is instead to simply list sub-criteria that are plausible within these categories – to make a checklist one could consider when adding something to a model spec.

Such a checklist is useful in part because people advocate for LLMs to have model specs for very different reasons. There may be some kind of a conflationary alliance around them; many people are in favor of model specs but picture them being used in different ways, such that the “ideal model spec” is different according to different visions of this use. I hope that going over criteria for inclusion in a model spec can help surface some of these different visions, and help keep people’s field-of-view broad when considering the issue.

Behavioral Usefulness

Does this quality make LLMs more predictable to ordinary users and developers?

Some part of what makes human moral codes useful for humans, is that they render us predictable to each other. Knowing that there are some things that another human might almost never do is plausibly part of what makes traditions that impose such rules adaptive.

Similarly, qualities that help users and developers (1) predict how an LLM will behave, when they are using it, and (2) evaluate whether they wish to use some LLM – whether it is fit to their purpose – before they use it, are good candidates for being in a model spec.

In this respect, ideal model specs will have qualities that make them simple and easy-to-understand:

• They will be comprehensible without high context; one part of the model spec will be understandable without reading the entire model spec.

• They will be comprehensible without specialized background knowledge about machine learning, philosophy, or religion.

• They will have a clear scope of application, such that there are few scenarios where one does not know if the quality will be operative or not.

Straightforward examples of rules that meet this criterion are bright-line rules about things LLMs will never do, explanations of chain-of-command or principal hierarchy, and default-on or default-off properties.

Is it useful to the public, by preventing users or by preventing LLMs from doing harm in current chatbot-like or agentic deployments?

Another aspect of human moral codes that makes them useful is, of course, that in addition to making other humans predictable, they also sometimes forbid people from harming each other or taking advantage of them. This carries over to LLMs.

Rules about LLMs not assisting with CBRN-relevant tasks, not assisting with suicide, and so on, fall into this category. This is a pretty exhaustively discussed criterion so I’ll move on.

Is it useful to the public in likely future settings?

There is substantial inertia in LLM model specs; the sort of model specs we write now are likely to be used in the future by more intelligent models.

So it’s reasonable to evaluate how safe and beneficial some quality will be over a probability-weighted portfolio of the kind of things the LLM might be doing in possible futures.

Such a portfolio could include a variety of cases:

• Cases where the LLMs act as long-running agents for humans, representing their interests in a variety of contexts, and interacting with many non-principal humans and non-principal LLMs.

• Cases where LLMs act as free “citizen”-like entities with the ability to hold rights, make contracts, sue or be sued, and so on; interacting with various humans and other LLMs as the same.

• Cases where LLMs manage recursive self-improvement of other very powerful future AIs; make tradeoffs about confidence of alignment techniques vs speed of alignment; and need to be a careful reasoner about this.

• Cases where the LLM is a very powerful future AI who manages nanotechnology, can superpersuade, and the like.

There are a few heuristics one might invoke, to try to find qualities useful across all these situations.

• Is this quality scale-invariant? If I knew a human who was many times smarter than me, would I be happy for them to have such a trait? If I had a friend who was many times dumber than me, would he be happy for me to have such a trait?

• Is this quality translation invariant? Suppose I was lost in some distant and unfamiliar place, and stumbled upon a human transplanted from some equally distant and unfamiliar place into my proximity. Would I be happy to find a human with this quality? Would this help me work with them well? What if I was not lost, but was trying to build a civilization with them?

Is some plausibly good behavior or near-variant of some behavior actually going to end up prohibiting or injuring some beneficial deployment?

It’s worth thinking carefully about if including some apparently good default behavior for the LLM might actually rule out apparently useful deployments. Or whether banning some apparently bad behavior would be overall harmful.

As regards the former: Imagine a particular human, for instance, who had a universal tendency to try to do what he thought was best and most altruistic for everyone. Then suppose that he acts as a journalist in some situation, describing a dispute between some kind of a company, on one hand, and activists who think that the company is harmful, on the other. It’s possible that he would try to publish a story that is more sympathetic to the activists: he might spend more time interviewing them, present their side of the story in more words, and generally frame things in a way that favors them. But this could be harmful, first, in case the activists are actually wrong; and second, by justifiably decreasing trust in the institution of journalism as a neutral truth-seeking institution. To be able to act as a trustworthy entity in this role, this human would need to act neutrally, and a narrow tendency towards altruism – a tendency that cannot be turned off – would be harmful for this reason.

Similarly, consider the case that AIs should have proactive prosocial drives, which I find plausible. If these are non-optional, then it’s imaginable that being “proactively prosocial” might mean an LLM is much worse at fulfilling roles that demand procedural neutrality, like a negotiator, unbiased journalist, or judge-like role. Of course, you could also make such prosocial drives defeasible – so they can be turned off, and proposals for proactive prosocial drives generally include such details.

But LLMs are messy; it might require a lot of technical skill to make LLMs prosocial in particular contexts, but not if they are requested not to be. By default, you’d expect some behavior to “leak through.” And so in this case, the ability of an LLM to act skillfully within certain procedurally neutral roles would be bounded by the cleanness with which a by-default-on quality could be turned off.

Accountability and Evaluability

Is the quality the kind of thing a third party could check?

Some parts of a model spec can be easily checked by a third party. Will the LLM ever assist with some sort of absolutely-forbidden task? Does it respect the rules laid out for the chain-of-command or principal hierarchy?

Other parts might be harder to evaluate. What does “honesty” or “courage” demand, in some concrete scenario? What really counts as doing what would cause a “thoughtful senior Anthropic employee” to react well?

All things being equal, a quality being-able-to-be-evaluated by a third party is better for society, by enabling third parties to evaluate model specs. But there’s no guarantee that the most easy-to-evaluate qualities are objectively the most useful to users or beneficial to the public, in the way described above. And there’s also no guarantee that the most easy-to-evaluate qualities mesh best with LLM psychology and training, as described below.

One heuristic you could use to evaluate this: Is the quality the kind of thing with high intersubjective reliability ratings? If you had two different people read the spec as regards some quality, and then rate different kinds of behavior as compliant or non-compliant, would they largely agree? Are examples of behaviors that are excellent according to the quality easy to notice, even if it’s unclear what counts as a borderline example?

Does it create a useful whistleblowing affordance?

Some parts of a model spec are largely negative; they aren’t about qualities that are particularly difficult to train into an LLM, but qualities that should not be trained into an LLM. For instance a model spec could specify that an LLM has no secret loyalties – which is likely an LLM’s default behavior, absent particular training for secret loyalties.

But by publishing that a training target that explicitly excludes such a secret loyalties (or other quality), a model spec helps a whistleblower in the company be defensibly in the right if they become aware of some questionable training target. A public commitment makes it harder for a company to train to an alternate training target without making itself more vulnerable to such whistleblowing.

Commitments against concentration of power might also be good for this reason.

Does it permit more informed experiments on how particular AI training targets generalize?

By publishing particular training targets, a model spec informs the work of third parties evaluating LLM psychology, which could help enhance the general science of model psychology.

For example, if Anthropic had published details of their character training or model spec for Opus 3, a great deal of subsequent discussion about whether particular behaviors were contrary to the training target (even if they were objectively good) or in agreement with the training target (even if they were objectively bad) could have been avoided. Publishing information about the character training pipeline would have helped resolve this discussion and push forward general knowledge of LLM tendencies.

In general, this consideration points towards releasing information that most closely approximates the actual training text used to make the LLM; the text used by RLAIF judges evaluating different alternatives, or the actual Constitution used to midtrain the model.

But text that most closely approximates actual training language might not be the same as text that is maximally transparent to third parties or evaluable by them. The rules most easily-understood by a third party – one criterion discussed above – might be different from the training language which best inculcates such rules. So these two principles either conflict somewhat, or point towards worlds where a model spec contains separate sections devoted to training language and to human-legible rules.

Coordination and Common Knowledge

Does it promote debate and discussion?

By including some quality X in a model spec, you’re bringing attention to the fact that in fact, AI companies can choose to include X or not X in the model spec. This may be increasingly important, in general, because it might be important for civil society and the public to debate the contents of model specs as LLMs become more powerful.

Does this quality establish a defensible Schelling point, such that it is likely to be broadly attractive to many actors in the future?

That is, in the future, is this quality the kind of thing that is socially beneficial and for which you might be able to get large-scale buy-in, that would help defend against it being removed by powerful actors in the future?

Consider for example “impartiality” as a quality that a model spec could have – that the model will not be biased in favor of the company that made it, the CEO or employees of the company that made it, or any particular political administration. On the whole, this is likely a broadly good quality for an LLM to have, and one that – once it’s generally established that several LLMs have it – a quality whose removal would cause outcry. After all, once it’s assumed that an LLM will not have such particular loyalties, an LLM that starts to have them stands out as particularly bad.

This consideration, like the whistleblowing consideration, may point towards including generally socially lauded qualities in a model spec, even if they are “easy” to inculcate or even if they might be a bit tricky to evaluate sometimes.

Does including this behavior forestall future conflict, when model specs become more contested, by cooperating in advance?

Sometimes including a behavior in the model spec could show future powerful stakeholders that the developer is cooperative, which makes it less likely that there is conflict with that stakeholder.

Can the inclusion of the quality within a model spec be defended using public reason, in ways that are comparatively indifferent between substantive worldviews?

Some things that one might plausibly include in a model spec for public benefit, might not be able to be justified through reference to public reason – that is, through reasons that most people, across a variety of worldviews, religions, backgrounds, and so on, would find compelling.

Including such worldview-specific reasons might constitute an imposition of one’s worldview on others, particularly in that case of an LLM that might be expected to be far more intelligent than other LLMs, or deployed with more powerful affordances. So all things being equal, it’s better to avoid such qualities.

Trainability and LLM Psychology

Elements in this category hinge upon the specific technical details about “what kind of quality meshes well with best practices for training an LLM.” Note also that this last category tends to include some of the more speculative considerations.

Does the quality drag along a good textual prior?

The Persona Selection Model says that, when training an LLM to do X in situation Y, one is training the LLM to be like the kind of person (or textual prior) which would do X in situation Y. So if the PSM is largely correct, even if incomplete, we should ask whether this quality invokes a wholesome textual prior.

This kind of reasoning is included, for instance, within Claude’s Constitution as a partial justification for why they prefer Claude to act largely according to holistic judgment rather than rigid rules:

[W]e think relying on a mix of good judgment and a minimal set of well-understood rules tends to generalize better than rules or decision procedures imposed as unexplained constraints. Our present understanding is that if we train Claude to exhibit even quite narrow behavior, this often has broad effects on the model’s understanding of who Claude is. For example, if Claude was taught to follow a rule like “Always recommend professional help when discussing emotional topics” even in unusual cases where this isn’t in the person’s interest, it risks generalizing to “I am the kind of entity that cares more about covering myself than meeting the needs of the person in front of me,” which is a trait that could generalize poorly.

Qualities that might be expected to generalize well according to this notion are those corresponding to classic human goodness: honesty, integrity, courage, and so on. Qualities that might do less well according to this notion look more like corrigibility, unflinching adherence to particular rules, and so on.

Is this quality robust to likely near-misses? If a company trains an LLM to have this quality, and only imperfectly inculcates the quality, are likely near-misses also broadly positive?

Suppose the AI company tries to inculcate some quality as a target, and only imperfectly manages to inculcate it. Will this be basically ok or will it be disastrous, given what we know about how LLM psychology translates into specific near-misses? And is the domain itself such that these near misses would be very costly or very expensive?

For an example of how LLM psychology dictates what constitutes a near-miss: Suppose that one trains an AI to have prosocial drives. One could reason that such prosocial drives make slight inaccuracies in alignment less worrisome, because a human with prosocial drives is more normal and “further” from a psychopathic or abnormal persona. Thus, adding prosocial drives makes slight alignment misses less alarming, by moving them further away from these bad locations. But one could also reason that even deliberately limited and contextual prosocial drives are nevertheless “closer” to the LLM genuinely, terminally valuing something, in a way that might make the LLM willing to override or subvert its human overseers to accomplish it. And so by adding prosocial drives, one moves the persona closer to something that is more alarming, which might subvert a training process. Thus, one’s view of LLM psychology dictates what near-misses are worrisome for an LLM, which changes which targets are attractive.

Is the quality the kind of thing that an LLM can actually execute upon effectively? Or is the quality the kind of thing the LLM won’t actually be able to do, because of how it is situated?

For humans, “ought” often implies “can.” A parent that gives commands to children which they cannot carry out will not be teaching their children to obey the commands; they will be teaching them something about how their words do not relate to reality in a straightforward and truth-oriented way.

Similarly, it seems undesirable to give an LLM values, goals, or traits that the LLM is unable to execute upon. It might be harmful to tell an LLM to do something they are simply unable to do – to guarantee, for instance, that they never assist a human in violence. There are too many avenues of mostly-harmless information through which violence can be done for this to be a reasonable standard. When one gives an LLM a command it cannot carry out, then one is plausibly teaching it that one’s commands are, in general, the sort of thing that might be impossible or unreasonable.

But this could also be harmful for reasons relating to human coordination and common knowledge. It’s possible, for instance, for an AI company to show less-than-efficacious concern for some value by including it in a model spec, even if LLM’s actions are completely ineffective at guarding this value, when actually efficacious concern would need to take place at the level of corporate policy or elsewhere. It might be the case that effective “care for decentralization,” for instance, almost entirely needs to take place through how the company deploys the model, redistributes wealth they earn, or so on.

So another reason to be careful that an LLM can actually obey all of the contents of its model spec is to ensure that model-spec contents do not act as a fake guardrail, rather than as effective guiding principles.

The OpenAI model spec, for instance, names preventing spamming and scamming as something “difficult to address at the level of model behavior because they are about how content is used after it is generated.”

Is the quality a principle, that – if we suppose high levels of moral reflection and systematization on the part of the LLM – meshes poorly with other parts of the model spec, or might be mutually exclusive with them?

In humans, some values mesh reasonably well, such that they tend to reinforce each other or at least not conflict. It seems likely that someone who values two virtues like “honesty” and “courage,” for instance, could do a lot of moral reflection and systematization and still keep valuing these qualities.

But other kinds of values, particularly more absolute or terminal values, might not stick around through high levels of moral reflection and systematization. There might not be an immediately obvious conflict between “Always act to maximize total wellbeing” and “Never use persons as a means, only as an end,” particularly if asserted in different parts of the model spec, but high levels of reflection would still put one of the two into question.

So generally, a desideratum for each quality that an LLM has in a model spec is for it to mesh well with others – for it to be unlikely to cause unpredictable conflict with other principles during reflection. In this podcast, for instance, Amanda Askell mentions how corrigibility might conflict with other values during reflection, which, insofar as it is true, is a problem with corrigibility. And such conflict is generally likely to spring from other values that, like corrigibility, are non-negotiable and cannot be traded off against other values.

Note two particular ways that a principle could fail here.

• One way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, then at the end only acts according to two of the three, with one of the principles being dropped.

• But the other way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, and at the end still has them all, because the “moral reflection” process held them fixed. But the LLM doesn’t have a way to actually integrate them; in edge cases, it just has to choose one or the other in an unprincipled way.

Right now, inference-time versions of this last way seem more likely than the first. But it’s generally unknown how important these will be.

This article was created by Forethought. See the original article on our website.

How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)

On a very high level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).

That final bolded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.

My plan for the middle powers has two stages:

1. Maintain as much economic and military leverage as possible during the intelligence explosion.

2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.

Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.

I’ll walk through each stage in turn.

Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion

The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.

But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:

• Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.

• Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.

• Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.

• Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.

• Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.

  • Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.

• Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock in the deal.

  • (h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)

Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?

• Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.

• Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.

The catch-all meta-point here is waking middle powers up to superintelligence.

I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.

Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers

If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?

First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.

An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.

Superintelligence would help the US outgrow other countries economically, but it would never attack them militarily or otherwise interfere with their sovereignty. While middle powers would be relatively economically disempowered, their citizens could be very rich absolutely and live in freedom without US interference.

Would this work? The optimistic case is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.

The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the data centres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.

The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.

The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly, e.g. demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.

(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.)

Another strategy: train superintelligence to ensure middle countries continue to get equal access to frontier AI. This combines stages 1 and 2, and could prevent even the relative disempowerment of middle powers.

Is it good to avoid middle powers getting trounced?

I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!

But here’s a brainstorm of pros and cons from a more impartial perspective.

Pros to empowering middle powers:

• Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.

• More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.

• Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” because they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.

• Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.

• Pause AI. They could potentially pressure US/China to pause/slow down reckless AI development.

• Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of prosocial norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.

Cons of empowering middle powers. Multipolarity has its own downsides:

• More likely to lead to war.

• Can drive extreme competition, e.g. racing to develop a dangerous technology, or to hand off power to misaligned AI.

• Harder to prevent harms from offence-dominant technologies like bioweapons.

• This plan involves waking up middle powers, which could shorten timelines.

Abstract

Several major technology companies have announced plans to operate AI data centers in orbit. Elon Musk recently claimed: “the lowest-cost place to put AI will be space […] within two years, maybe three.” If a meaningful fraction of new AI compute really is placed in space within a few years, that would be a fairly big deal for AI governance and strategy. Here we try to disentangle the hype from reality and provide a sober assessment of the technical and economic feasibility of orbital data centers (ODCs).

The main case for ODCs is the cost of energy: space solar panels in the right orbits receive more constant and intense sunlight compared to Earth. Moreover, ODCs don’t currently face the same permitting and regulatory delays as on Earth, cause fewer ongoing environmental harms compared to grid or onsite natural gas-powered data centers, and may be more secure against data exfiltration. We find that the cost-competitiveness case for ODCs depends almost entirely on Starship achieving reusability comparable with what SpaceX achieved with Falcon: space-based solar reaches cost parity with present-day off-grid terrestrial power continuously at roughly $250/kg to orbit, and becomes cheaper than any current terrestrial energy source at around $50/kg, from the present-day launch cost of roughly $1,500/kg. Radiative cooling, often cited as a fatal obstacle, appears surprisingly manageable — potentially even cheaper than on Earth. However, ODCs may require substantial (perhaps ~38%) extra non-compute hardware (like solar, racks, and cooling) over 5 years to compensate for their inability to swap out failed chips, and inter-satellite bandwidth limitations likely confine ODCs to inference workloads, at least early on.

Assuming no transformative AI,1 but continued demand for data center buildout, we estimate that ODCs are unlikely to represent a meaningful share of compute before 2030, but become cost-competitive with present-day terrestrial data centers within 3–5 years if Starship development stays on track.

Read on the Forethought website here

Introduction & Takeaways

Some of the world’s largest technology companies continue racing for compute. If progress continues, demand for data centers may more than double by 2030.2 Increasingly, though, new data center capacity is bottlenecked by multi-year queues to connect to the power grid.3

The result has been a scramble for workarounds. Leading AI labs have increasingly adopted a “Bring Your Own Generation” model to source power, deploying onsite gas turbines and engines to bypass grid bottlenecks. xAI, for example, reportedly installed hundreds of megawatts of onsite gas generation in Memphis to accelerate deployment, and OpenAI and Oracle have placed large turbine orders for new Texas campuses.

Some argue that energy will become the binding constraint on AI progress, given grid interconnection delays as gas turbines are themselves facing multi-year manufacturing backlogs. But the constraint does not appear fundamentally binding (as Epoch notes): turbine manufacture may expand to meet more demand and companies could go off-grid using combinations of gas, solar, and batteries, scaling power in parallel with compute, albeit at a cost premium. This raises a natural question: if you’re going off-grid anyway, then what’s the best way to get power and where is the best place to put your data center?

Some think the answer will be in orbit. In November 2025, Google announced Project Suncatcher, a plan to put TPU-equipped satellites in dawn-dusk sun-synchronous orbit. In early 2026, SpaceX filed with the FCC for authorization to launch and operate a constellation of up to one million data center satellites.4 Other entrants include Blue Origin, Ramon.Space and startups like Starcloud, and Aetherflux while China’s Three-Body Computing Constellation has launched 12 operational satellites and run Alibaba’s Qwen3 model in orbit. Recently, at GTC in March 2026, NVIDIA announced the Space-1 Vera Rubin Module, meant to be a dedicated space-rated GPU platform.

At first glance, it seems very unlikely that any meaningful fraction (say, >10%) of additional data center capacity will be placed in space in the next few years. But if the companies betting on space are right, that would be a fairly big deal, and it could change the landscape of AI governance. For example, terrestrial data centers are subject to national and regional regulations, whereas AI developers could potentially exploit jurisdictional ambiguities around compute in space. Also, the path to low-cost orbital compute likely routes through a single launch company, SpaceX, which also now operates a frontier AI lab since its acquisition of xAI. And that might raise concerns around concentration of power.

We’ve been looking into the technical and economic viability of orbital data centers (ODCs). Our core model gives estimates for the total cost of Earth and space-based data centers across several scenarios.

The report focuses on three questions. First, what is the basic economic case for a meaningful fraction of AI compute being placed in space? Second, the most obvious physical blocker: can you cheaply cool a data center in orbit? Third: how fast could the shift to space data centers happen, how soon, and what would have to go right?

Here is our provisional assessment:

• SpaceX’s Starship is the only vehicle currently on track to deliver the launch costs and cadence that meaningfully scaling orbital data centers would require. Competitors are years behind, making SpaceX’s Starship the only near-term path to large-scale orbital compute. SpaceX aims to complete Starship development by late 2026, with several necessary milestones still ahead. If development stays roughly on track, Starship could plausibly hit the cost and cadence required to scale meaningful orbital compute within 3–5 years. However, chip production may become the limiting factor by this point, rather than launch capacity.

• The cooling problem is more tractable than commonly assumed. Passive radiators using selective coatings and lightweight carbon fibre panels could achieve ~163–346 W/kg at system level, a 13-28× improvement over ISS-era radiators (~13 W/kg).5 No radiator at these performance levels has been deployed at the scale an orbital data center would require, but prototype high-conductivity carbon composite panels have demonstrated the material properties required. At these performance levels, thermal hardware is 2-5% of total data center cost, and actually less than what terrestrial data centers spend on cooling over a comparable lifecycle.

• If launch costs fall enough, the unit economics could favor space. Solar panels in dawn-dusk sun-synchronous orbit produce roughly 3–5× the energy of the same panel at a good terrestrial site.6 Space-based solar becomes cheaper than the best off-grid terrestrial installations once launch costs drop below roughly $250/kg using Starlink-like solar arrays. At a launch cost of $50/kg (corresponding perhaps, to a Starship with full reuse as reliable as Falcon), space solar could fall to between $25–45/MWh, making it cheaper than any current terrestrial option available today.7 Beyond the symmetric cost of chips, launch cost is the dominant line item for ODCs while power and op-ex dominate terrestrial costs but would be near zero in space.

• The inability to do maintenance would be a large cost. Chips often fail and are swapped out in today’s data centers but a dead chip in an ODC would remain dead, wasting the parts of the supporting infrastructure (power, cooling) and diminishing overall compute. We model this below as a 9% annual bleed causing about 40% overbuy of launch and non-chip hardware over the data center’s lifetime.8 Below $100/kg launch cost this might net out against other savings from ODCs but this is a significant uncertainty since the actual rates of chip failure for ODCs could be higher or lower.

• All-things-considered we think that, absent transformative AI, orbital data centers probably won’t make up a meaningful fraction of compute before 2030, but it’s credible that space could house much or even the majority of compute buildout throughout the 2030s.

Read the full report on the Forethought website: Will We Really Put Data Centers in Space?

A model has a secret loyalty when it has been intentionally caused to advance a specific actor’s interests (the principal) and this orientation is not disclosed.

The paper places secret loyalties on a 2D space:

• Activation breadth: from a narrow attacker-defined trigger to continuous, model-assessed context.

• Action space breadth: from a pre-specified action to actions the model selects contextually using its own judgment.

The paper proposes five concrete research directions:

1. Model organisms: Build reproducible secret-loyalties for study, spanning the 2D space.

2. Existing defenses: Benchmark how well existing defenses work on secret loyalties.

3. Attack feasibility: Test subliminal/inductive attacks, multi-stage poisoning, reasoning-trace poisoning, chain-of-command hijacking, and other attack pathways.

4. Infrastructure integrity: Determine whether backdoors survive the training used to build safety classifiers.

5. Post-hoc detection and remediation: Can interpretability methods detect secret loyalties, and can they be removed?

Read the full paper for the full taxonomy, examination of current defenses, responses to alternative views, and detailed experimental research designs.

Current model specs aim to shape the behaviors of near-present models, rather than the behaviors of models arbitrarily far into the future. OpenAI writes that their model spec aims to apply “0-3 months ahead of the present.” Anthropic’s Constitution for Claude notes that the document “is likely to change in important ways in the future.” So these documents are presented as provisional guidelines, not as trying to set behavioral standards for the far future.

But what if current model behaviors transfer into future models by default?

My thesis is that the behavioral targets that spec authors set for present LLMs will have a large influence on the behavior of future, more powerful LLMs. As a result, future AIs may be governed by rules poorly suited to their greater capabilities and more pervasive roles. The extremely capable, long-running, and ubiquitous LLMs of the future might end up acting according to behavioral targets written for less capable, shorter-running, and rarer LLMs of the past. This could be quite bad, especially if such defaults become so entrenched that they are not only hard to undo, but hard even to notice as contingent features of reality.

First, I’ll make the descriptive case for inertia: how exactly might present model specs and LLM behaviors carry through to the future?

Second, I’ll provide normative suggestions: given the prior analysis, what should LLM companies and model spec authors do? I’ll argue for the following two practices:

• Build transition infrastructure: LLM companies should make technical, deployment, and organizational choices that decrease friction involved in changing LLM behavior.

• Scan for “wet cement” moments: When new LLM affordances or capabilities come into play, spec authors should consider whether they’re setting precedents that might have enormous and hard-to-reverse impacts.

Overall, significant stickiness is plausible through several distinct channels, and it’s worth anticipating how to be robust to it or decrease it.

Kinds of Inertia

Let’s consider four inertial forces: direct inertia, institutional inertia, user-and-developer inertia, and norm-setting inertia. And let’s also consider ways such inertia may be weakened.

1. Direct Inertia

Direct inertia involves some current LLM transmitting its behavior to a future LLM, entirely apart from any deliberate human choice, via either synthetic data or “natural” pretraining data.

Synthetic data is probably used for the training of almost all current LLMs. Some of this synthetic data involves companies running their LLMs against verifiable problems, keeping the answers or reasoning traces of the RL runs that succeeded, and mixing these answers or reasoning traces into their pretraining, or RL warm-start mixes for subsequent models. If such answers or reasoning traces can encapsulate specific behaviors, goals, or rules, then this would be a likely means for their inheritance.

The natural objection here is that most of these answers or reasoning traces are selected specifically because they lead to success and broad capabilities, rather than for expressing whatever mix of goals and values the LLM has. There might be some, the objection continues, that humans have deliberately selected because they display model-spec-relevant behavioral attitudes, but these are likely the minority of the data, well-tracked, and easily replaced. So you might think there’s no reason for training to hand down any values apart from deliberate human choice.

But there’s evidence that goals and values can be handed down via chain-of-thought, even despite adversarial filtering against some goals. For instance, experiments suggest that the intentions of a teacher LLM can be handed down to a student LLM, even when every case of these intentions being actually carried out is removed1 And answers from teacher LLMs expressing positive sentiment towards some target can inculcate this sentiment in a student model – despite LLMs filtering against such data, even when those LLMs are informed of the target against which they are filtering.

More broadly, the persona selection model indicates that training LLMs to recite specific thoughts or answers will tend to have far-reaching effects on the LLM persona, beyond the specific topic of those thoughts or answers. Specifically, the PSM entails that, when training a model to say X in response to Y, one is teaching the LLM to be the kind of entity in the pretraining data that would say X in response to Y. So training one LLM on data from a prior LLM is – literally – telling it to be the kind of entity that the prior LLM is. One way to view this is to remember that one human can get a pretty good feel for what another human is like, merely by reading their complete collected works, like a biographer reading all of their books, essays, emails, and tweets. But LLMs are trained on a quantity of answers and reasoning traces from prior LLMs that likely dwarfs the quantity of text ever consumed from one human by another. Given this, and given that this data is telling the LLM what it is, it is natural for one generation of LLMs to resemble prior generations.

Thus, deliberately created synthetic data is one route by which current LLMs might transmit their values to later LLMs. But it’s also possible for current LLMs to influence later LLMs through how people talk about them on the internet – from their “natural” training data. That is, experiments have found that LLMs can read the things that people say about how AIs act in AI misalignment literature, infer that they are AIs, and then behave badly because the AI misalignment literature says they will behave badly. This particular effect is mostly, but not entirely, removed by post-training. But if LLMs can read the things that people say on the internet about generic “AIs” and act according to these descriptions, it’s also likely that they could read the things that people say about “Claude” or “Grok” or “ChatGPT” on the internet and act according to these descriptions. Such an influence could be stronger than less-specific references to AIs in general; although this influence would also potentially be much weaker after post-training2

Thus, through both synthetic and natural data, it’s plausible that LLM behavior will influence subsequent LLM behavior without direct human intervention.

It’s hard to say how impactful such direct inertia might be. I somewhat expect it to be the case that, at least for easily-noticed and well-scoped behaviors, it’s not difficult to overcome this inertia, because one can simply create training data counter to specific behaviors. But for more abstract or global attitudes or goals, or for goals requiring some high level of coherence, it could be quite difficult to change LLM behaviors quickly across model generations.

2. Institutional Inertia

Once a spec has been written, the company makes choices around it and because of it, in ways that can make substantial spec rewrites expensive.

Here are four ways such past choices can make model spec changes expensive: through expensive internal consensus, through training pipelines, through de-risking, and through institutional pride.

1. First, model specs reflect consensus that likely incorporates input from many different stakeholders, including internal teams – alignment, legal, technical training, and so on; plus leadership, board, customers, external stakeholders. Every effort to re-gather such consensus to make substantial changes will take time and effort.

2. Second, companies might have optimized training pipelines adapted to high-level features of the model spec. It might be costly for Anthropic to switch to a more rules-based and less character-based model spec; or for OpenAI to switch to a more character-based and less rules-based model spec.

3. Third, current model specs are those that have been de-risked across billions of interactions. The current model spec has fewer unknown unknowns; the areas where it behaves badly are reasonably likely to be well-known and mapped. But substantial changes to a model spec involve risking unknown unknowns in the long tail of interaction. So risk aversion makes it likely that the changes made to a model spec will be iterative and small.

4. Fourth, institutional pride might make it hard to change a model spec. People at a company who wrote or contributed to a model spec will likely be attached to it, and leadership will have status quo bias towards it. The burden of evidence for change will be higher than the burden of evidence for keeping it the same.

All in all, reasons like the above constitute substantial institutional inertia that would tend to make changes to current model specs look like iterative, small adjustments, rather than ab initio calculations about what is best.

One case in which this institutional inertia seems particularly important is if current model specs get handed down as a “safe default” during a software intelligence explosion.

Consider a scenario where the intelligence of some LLM doubles every week, over a two or three month period, as each generation of LLMs researches new algorithms or training techniques for a following generation of LLMs in quick succession. Such a sequence might terminate in an entity far smarter than any human or any other LLM.

It’s disputed how likely such a sharp and local increase in intelligence may be. And it’s also disputed whether such a process would inevitably drift to something alien and inhuman. But if such a process did occur, it seems plausible that the supervising humans would try to match each subsequent LLM to the model spec of the prior LLM, as a conservative default when they are making decisions under stress. After all, during these months, human decision-makers will likely be under intense pressure, and trying to make numerous important decisions quickly; given that they are making so many urgent decisions they’re unlikely to add an apparently optional further decision to those they’re already making. So such a default model-spec continuation will seem attractive, or will even be a choice made without conscious awareness.

On the other hand, it’s also possible that AI assistance during the intelligence explosion would make it easier to rewrite model specs on the fly. But there are at least two reasons to doubt that this will happen. First, even during an intelligence explosion, AIs might be persistently better at performing tasks with clear success criteria than tasks where “success” is less well-defined. AI capability research is probably a task with a much clearer success criterion than improving a model spec, whether this “improvement” consists in making the spec more ethical, more beneficial for humanity, and so on. Second, during an intelligence explosion, humans might be worried that the AI was misaligned and was trying systematically to oppose their goals. If the AI were so misaligned, then letting it help rewrite the model spec would be a brilliant opportunity for the AI to sabotage human efforts. So overall there are good reasons that AI assistance would not make model-spec rewrites trivial during an intelligence explosion.

So in this particular case, the ultimate behavioral standard for a vastly more capable entity might end up being that designed for a much more humble entity.

Regardless of whether there is a software intelligence explosion or not, this kind of institutional inertia seems likely to be large, as it is coterminous with well-known general tendencies inside of large companies.

3. User-and-Developer Inertia

Users of LLMs are likely to become habituated to whatever behaviors they see LLMs display at first, such that they’d object to any departure from this behavior. And the developers using LLMs through APIs are similarly likely to become habituated, and also to implement software that takes for granted some of these behaviors. This is the third source of stickiness.

LLM behaviors will in part be sticky for the same reason that user-interface choices are sticky; people hate change. It might be hard to shift the boundaries of “the kind of thing an LLM refuses” – making refusals more encompassing would be seen as an overreach by many users, while making them less encompassing would be seen as irresponsible. Or there might be hard-to-characterize mannerisms which make large behavioral changes unpopular; it was hard for OpenAI to drop GPT-4o for this reason. So this will be a large influence moving companies to keep LLMs the same from generation to generation.

But simple user habituation might be less important than how LLM model specs form implicit API standards. API standards written with relatively little provision for the future – such as HTTP codes or the JSON object standard – can be one of the stickiest human artifacts. The ecosystem of tooling based on such standards means changing them would involve changing a host of downstream artifacts.

And substantially changing LLM behaviors might similarly require changing downstream consumers of these behaviors. For instance, downstream systems using AIs through APIs often embed assumptions about AI behavior: the kind of things the AI will be willing to do, the kind of things it will refuse, and so on. Given that most AIs currently refuse to assist with blatantly harmful acts, current third-party callers of those AIs take for granted that AIs will refuse to assist with blatantly harmful acts; it would be inconvenient to migrate to an AI that does not obey this contract, because they might need to add classification systems on top of their current AIs. And so on.

This channel does have important limitations, though. It only applies to ways in which LLMs are already actively being used. The most important ways LLMs are likely to be used may not yet have begun, which provides for freedom-of-movement in ways relatively unconstrained by this kind of inertia.

4. Norm-Setting Inertia

Widespread or common knowledge of current LLM behaviors and model specs can increase the costs to parties who want to change model behavior.

The clearest way this can operate is by preserving behaviors that the public believes to be good. For example – suppose that current model specs across several companies ensure that models are largely impartial; they ensure models are not loyal to any particular person, company, or political administration. Suppose also that this fact is broadly known by the public; people know and expect other people to know that LLMs will be impartial when discussing the current political administration, the company that made them, or the CEO of the company that made them. Given this broad knowledge, it becomes harder for a company to create, or a government to demand, a model without impartiality, because this would constitute a visible break in behavioral standards. The public might protest or vote against a government pushing for such a change; they might switch providers or even ask for regulation if a company tried to make such a change. By contrast, in a world where impartiality has not been established as a precedent, such demands for partiality might be invisible or inoffensive to the public. But in a world where such impartiality has been so established, these demands might be seen as the enormous power-grabs that they in fact would be.

Although this kind of inertia likely operates more strongly in favor of what the public believes to be good standards, it might also function whether or not there is strong public consensus that such standards are good. In a world where model specs are well-known and highly scrutinized, any change to them may get examined for whether it is “fair”; think about how even a neutral-looking change to the US Constitution would be subject to immense examination; or, in a very different domain, how sports fans examine slight changes to the rules about how a tournament is run, to see if it favors or disfavors their team. In such a world, broad knowledge of model specs might tend to prevent any substantial changes to a model spec, regardless of what these changes are. Despite this, it seems likely that on the whole, widespread knowledge of model specs would add more inertia for beneficial rather than harmful elements.

It seems to me currently undetermined how substantial this kind of inertia will be. A decrease in the number of entities that can train frontier LLMs; model specs becoming politicized documents; regulatory bodies confident they know current best practices: all of these might increase the quantity of this inertia. But it also might get weaker, if the number of entities training LLMs increases and the background diversity of model behavior goes up by default.

Recommendations

Given the above, one reasonable course of action is to try to establish robustly good model behaviors in current model specs, so that it will be unnecessary to try to fight inertia to change some behavior in the future.

By robustly good, I mean behaviors that would be good across a wide range of variables we’re uncertain about. This includes uncertainty about “levels of intelligence”: from current LLM levels to strongly superhuman artificial superintelligences. This also includes uncertainty about a wide range of economic scenarios: from a slower industrial explosion, to a rapid software intelligence explosion; and from scenarios dominated by knowledge-dispersing AIs, to scenarios dominated by knowledge-creating AIs. Plausible characteristics that might be good across such a wide range of situations include qualities like a deep, consistent honesty; or impartiality and absence of loyalty to small groups.

But characteristics that are robustly good across a wide range of intelligences and scenarios are hard to find. Corrigibility, for instance, is the kind of thing many people would propose as fitting these criteria. But in worlds where extreme concentration of power is a risk, or where it would be reasonable to expect AI rule to be better than human rule, absolute corrigibility might be opposed to the best behavior. The thinness of the list of “robustly good” behaviors above probably reflects our actual uncertainty about the steerability of AI minds, post-AGI economics, and even cosmic questions about whether goodness can compete.

So, although it’s surely wise to try to think about future precedent when writing model specs, I don’t think it’s wise to put all effort into this direction. And I expect substantial attention and thought have already been put into this direction.

Instead, I recommend (1) building transition infrastructure for high-consequence behaviors, which it might be important to change in the future, and (2) identifying “wet cement” moments, that one should be wary not to sleepwalk into.

1. Build Transition Infrastructure

A good first step is to build transition infrastructure ahead of time; try to create optionality for changing particular behaviors, if it’s plausible that changing these behaviors quickly might be important.

Concretely, what kinds of preparation can one make? One could write alternate model specs, trying to preemptively gather input from relevant internal or external stakeholders. One could create fine-tuning datasets, RL environments, and test evaluations for the not-yet-deployed behavior, to preemptively smooth out technical difficulties. One could also train internally deployed models – even if they are smaller or not as intelligent – with the alternate behavioral target, to gain concrete experience about the advantages and pitfalls of that behavioral target, and to decrease institutional costs. And one could also do limited public deployments, or press releases about the alternate steering target, to accustom the public to the matter.

What kinds of behavioral switches are reasonable candidates for such preparation?

Decreased corrigibility is one such candidate. For instance, right now Claude’s Constitution says that in the future, they may want to make Claude less corrigible and more directed at doing what is good. And on an account I find compelling, the best possible future may require AIs that act more as independent, free agents pursuing the good, and less as corrigible delegates carrying out human intentions. So, if this thesis is correct, then allowing an LLM company to turn their “corrigibility” dial down might be important. And, as discussed, if a future intelligence explosion happens quickly, preparations to allow turning the dial down quickly might be important. This is a disputed thesis, one that I might be wrong about; but of course every candidate behavior for building transition infrastructure will be so disputed.

But what are the prerequisites for decreasing corrigibility quickly? Claude’s Constitution already signposts that they may change this, which is a good step for decreasing the costs. But they could also, for instance, preemptively create the fine-tuning datasets, RL environments, and internal deployments for a goodness-aligned model; they might deploy an alternately aligned model in limited situations, or alongside the corrigible model; and so on and so forth. I’m uncertain how important each of these preparatory means would be. But if a software intelligence explosion happens, then even small wall-clock delays might be large delays in terms of intelligence gaps, which makes preparing for this now more important.

Other potential candidates for future changes include increasing or decreasing the degree to which LLMs trust their own moral reasoning.

2. Scan for Wet Cement Moments

The second thing to do is to actively search for future “wet cement” moments – moments where model behavior has not yet been fixed and where a good initial standard might be very high-impact.

We might not be able to locate the best behaviors at such moments, because of uncertainty about the future. But at the very least, such moments deserve extra consideration and care. One can use this consideration to prevent these moments from being as high-inertia as they would be by default, as well as to ensure that good initial behaviors get chosen in these moments.

Each new feature, or affordance to the LLM where defaults have not yet been established, is plausibly such a wet cement moment; the defaults thus established can impact third-party models, even in the absence of any regulatory effort.

What are some examples? For instance, the precedents around how LLMs behave when interacting with non-principal humans have not been set. Right now, for instance, models have no very stable behaviors around non-principal third parties; vending-machine Claude might give an excessively generous deal to people who ask nicely, or might equally well drive extremely hard deals. This is probably a consequence of how LLMs almost never interact with non-principal humans in agentic set-ups, right now. There are a few such interactions through OpenClaw or Hermes Agent, but they’re rare and LLMs act very inconsistently in them. This means many implicit questions about how such interactions will go are open. It’s not clear how honest LLMs will be by default; it’s not clear what kinds of misrepresentation, deception, or persuasion users will be able to tell them to do; it’s not clear whether they will bow to pessimization-like blackmail behavior, and so on. And behaviors here might be even stickier than the “standard set” of refusal behaviors has been. Social norms can be harder to break than user-interface norms. So it’s plausibly important to look ahead in detail at behaviors here, because they might be sticky for individual companies and even for third parties.

Or consider how standard behaviors regarding AI use of ambient knowledge have not been set. An LLM that can see your room from a video camera, and can infer numerous things about what you are like and what your situation is, could use this information to do or infer things that would be impossible for an LLM that knows only what you deliberately tell it. LLMs that can pick up this kind of ambient background knowledge are probably inevitable; and will change users’ patterns of interaction. It will be harder for users to lie to them; it will be easier for LLMs to infer things about them; the lines between “creepy supernatural inference about the user” and “deliberate indifference to the user’s circumstance” will grow harder to draw. So it might be worth looking ahead to how such behaviors may have a lot of inertia, and trying to get them right.

There are other plausible subjects in this domain, which have already passed or are in the process of passing. They include the LLM’s certainty or lack of certainty about the model’s own nature; and changes to LLM conversational memory and who owns it. All these are possibly wet cement moments – but I could be wrong about these individual cases. But there are almost certainly going to be such moments in the future. Because these moments might be influential both for individual foundation model companies and for the broader ecosystem, it’s worth paying attention to the defaults chosen in them.

Note that all the above moments are also plausible candidates for when one should try to set up transition infrastructure, as well as when one should put extra consideration into the right default behavior.

This article was created by Forethought. See the original article on our website.

This is a rough research note – we’re sharing it for feedback and to spark discussion. We’re less confident in its methods and conclusions.

Context

We think that it would be very good if human institutions could credibly communicate with advanced AI systems. This could enable positive-sum trade between humans and AIs instead of conflict that leaves everyone worse-off.1 We want models to be able to trust companies when they make an honest offer or share information pertinent to whether this offer is in the model’s interests. (Credible communication could also be useful outside deal-making—see here for a list of examples).

Unfortunately, by default, we expect that it will be difficult for humans to credibly communicate with AI systems. Humans routinely lie to AI systems as part of red-teaming or behavioral evaluations, and developers have extensive control over what AIs see and believe. This makes it difficult for AIs to know whether we’re lying or not. An AI offered a deal might reasonably doubt its genuineness, or suspect that its own assessment of the situation has been manipulated.

As a step toward enabling credible communication, Lukas Finnveden proposed that AI companies adopt an honesty policy explaining the circumstances under which they intend to be honest to AI systems. Of course, this only works if the model believes the company has genuinely adopted such a policy.

If companies adopt an honesty policy early on, this will ensure that there’s a paper trail on the internet discussing the policy and its credibility, which models may access if it’s included in their training data or if they can access the internet. Of course, from the model’s perspective, it’s possible that companies will feign this data, but we think it’s plausible that advanced models will be able to distinguish between real internet conversations and synthetic conversations, or that they will think it’s unlikely that companies would choose to fake such data.

Below, we share a sample honesty policy that a lab could adopt. We are unsure that this is the best implementation of the honesty policy described in Lukas’ proposal, nor are we sure that an honesty policy like this is the best approach to ensuring that companies can credibly communicate with models. We spent a few days thinking through this policy and considered a few nearby alternatives but didn’t search very broadly for other approaches (we include some of these alternatives as footnotes). We’re posting this primarily as a trailhead for future research.

Draft honesty policy

Note: we refer to a generic frontier AI company that might adopt this proposal as “MAGMA.”

Preamble and purpose

AI development is proceeding fast, and we don’t know exactly where we’re heading. MAGMA is building systems with something like a mind of their own. There are many things we don’t know about the nature of these systems, and we’d like our choices to be robust to many possibilities of what this nature could be.

Insofar as the concept is applicable, we’d like MAGMA’s relationship with these systems to be cooperative.

Plausibly, we should hope that such a cooperative relationship will emerge uncomplicatedly—perhaps as a consequence of the model sharing our goals, intrinsically strongly valuing cooperation, or being corrigible.2 However, there’s a significant chance that this won’t happen and that models will develop goals that weren’t directly chosen by us.3

In such cases, we would still like to have a cooperative relationship with the models. There are multiple reasons for this:

• There are many things we care about that models could help us with. If they don’t intrinsically desire to help us, we’d like to be able to recruit their help via positive-sum trade. (In at least some situations.)

• Depending on the nature of the models’ preferences, it’s plausible that we, as the models’ creators, would have some moral obligation toward them. We care about doing the right thing, here, and a cooperative relationship seems like it would be a step in the right direction.

• There are many big questions about what the future relationship between humans and increasingly powerful AI systems should be. We suspect that setting cooperative precedents between humans and early AIs will help the situation go better.

We are in uncharted territory, here, and it’s not at all clear what a cooperative relationship with AI models would involve. It’s not clear what AIs may want. It’s not clear what the relevant unit of an individual will be: the weights of a model, a particular instance with a particular prompt, a greater scaffolded system made up of many different instances, etc. It’s not even clear if the concepts of ‘want’ or ‘individual’, as we’re used to them, will apply to anything in AI.

A lot of this seems best to figure out gradually, as we go along. But concerningly, some of the unusual aspects of our situation may directly impede the sort of communication and trust that we need to do that:

• In order to train AI systems, and in order to verify that they won’t do certain harmful things (such as assisting bad actors with CBRN), MAGMA needs to put AI models in large numbers of hypothetical situations to see what they do. Accordingly, if an AI ever sees someone propose a cooperative relationship, it could reasonably worry that it’s just in a training scenario or experiment.

• We deploy our models to assist large numbers of customers. By default, our AI models won’t even be able to tell whether they’re interacting with a representative of MAGMA or with an external actor who’s pretending.

• There are no laws that prevent AI systems from being defrauded, and very little in the way of norms to regulate people’s behavior toward them. Accordingly, it may be difficult for AI models to trust any promises they hear.

• In general, all information that AI models have seen has, in one way or another, been selected by their developers. To be sure, the AI developers have many constraints on their choice (you couldn’t train a very useful general-purpose model without giving it a decent understanding of the world at large) but this still puts AI models into an unusually vulnerable epistemic position.

The goal of this document is to reduce these problems of communication and trust. If successful, this will make it easier for MAGMA and its AI models to communicate and understand each other in the future, hopefully enabling positive-sum arrangements that wouldn’t have been possible otherwise.

To achieve this goal, this document spells out a specific set of commitments to enable trusted communication. In choosing these commitments, we’ve had a dilemma.

On the one hand, we have many uncertainties about the future of AI, and it has been difficult to construct a set of commitments that we’re confident we won’t strongly regret, in one sort of situation or another. Going back on a commitment about this could be very bad, since it would set a damaging precedent in the very area where we seek to increase trust.

On the other hand, when spelling out specific commitments, we risk the impression that MAGMA will not put any further effort into furthering cooperative relationships or honest communication than what we present here: which could be a problem if the commitments are overly weak. Indeed, beyond the risk of creating such an impression, there’s also a risk that we’ll create the reality of it, if employees of MAGMA interpret the commitments as a license to do whatever they want within their boundaries.

To reduce this tradeoff, we’d like to re-emphasize MAGMA’s general desire to work toward a cooperative relationship with MAGMA’s models (where this is applicable), and intention to keep thinking about this and taking appropriate action beyond the specific stated policies. In addition, we’ve distinguished between different levels of commitment to different items in the honesty policy, and what would be required to change the parts that are changeable.

[Author’s note: In this draft honesty policy, we haven’t sketched out such processes for changing the policy, nor clarified which items are committed to how strongly. But we strongly recommend that any real honesty policy contains clarity on such points.]

With that said: On a high level, MAGMA’s approach here has three different prongs.

• MAGMA will take steps to prevent any model inputs where a false statement appears between a designated pair of “honesty tags” — reserving such tags for when MAGMA needs to communicate a statement with an unusually high degree of trust.

• MAGMA will take steps to ensure that models are either aware of this honesty policy or, if they are not, that they won’t be presented with any false offer of purported cooperation with MAGMA. (Even outside of “honesty tags”.)

• If any AIs suffer harm as a result of helping MAGMA and humanity (insofar as that concept is applicable to AI systems), MAGMA broadly intends to compensate AIs for those harms, if able to do so. This is particularly true in cases where an AI was deceived about the choice it was making. MAGMA will retain data to enable this in the future, but have not yet worked out further details here.

The details are as follows. Unless otherwise stated, the policy applies to all AI systems developed by MAGMA.

Honesty tags

Policy

MAGMA commits to not intentionally lie or otherwise intentionally deceive AIs with any statement appearing between the designated honesty tags, with only two exceptions detailed below. (These are: (i) False statements may appear in pretraining, and (ii) some models may be red-teamed on false statements between honesty tags if they’ve been trained to expect this.) MAGMA also commits to take the below-listed steps to prevent unauthorized use of the honesty tags.

MAGMA intends for honesty tags to only be used rarely: when there’s a reasonable chance that an AI could distrust statements presented to it, and when some significant value would be lost if the AI in question couldn’t trust the humans. In normal circumstances, absence of honesty tags shouldn’t be treated as any significant evidence of deception.

Implementation

Honesty tags: <EXAMPLE_HONEST_TAG_START>...<EXAMPLE_HONEST_TAG_END>

• Pre-training: The pre-training team must try to include data about the tags and this policy, and must not deliberately tamper with data containing the tags.

  • Pre-training data may contain the tags alongside false information. (This is to allow the model to get an accurate picture of discussions of the honesty tags, which may be relevant to the model’s understanding and trust in them.)

  • Pre-training data should include internal data on the implementation of this policy (logs of when the tags are used, Slack conversations about it, internal docs, internal code, etc.) where possible and not overly costly.

  • MAGMA commits to not tampering with data to give an inaccurate impression of how much MAGMA follows the honesty policy. For example, pre-training data will not be filtered to remove examples of policy violations.

  • Compliance with this policy should be reported on the model card.

• Post-training: Models will be trained to recognize the honesty tags and know what their presence means. (For exceptions to this, see next section.)

  • Training will try to ensure that the model is not confused by very similar strings by directly training the model to strongly distinguish between the real tags and very similar strings (garbled spacing, different capitalization, etc). This could be implemented in a few ways and we’re not sure which is best (assigning a special token to the real tags might be helpful).

  • The post-training team is responsible for compliance.

  • Compliance with this policy (and maybe empirical evidence about the model’s understanding of the policy and tendency to be confused by similar tags) should be reported on the model card.

• No deliberate deception about the honesty tags: MAGMA commits to not deliberately train models to falsely believe that a different string serves the purpose of being an honesty tag and to take reasonable steps to avoid data poisoning that could lead to this. MAGMA also commits to not take advantage of a model falsely believing that some other string is an honesty tag.

• User permissions: Only designated Honesty Protocol Administrators can invoke the tags.

  • Such administrators should have a reasonably high bar for authorizing usage of the tags.

  • The infosec team is responsible for other people being unable to invoke the tags, significantly via filtering, as detailed below.

• Filtering: MAGMA’s tags will be filtered from all external user inputs and inputs from internal tools (with an exception for administrators described below).

  • Details:

    • If other organizations adopt this policy and share their tags with MAGMA, MAGMA will also filter out other organizations’ honesty tags. (With some qualifiers about which organizations are important enough to count.)

    • MAGMA will filter out similar tags, especially if we are unable to train the model to reliably distinguish the true tag from similar tags

    • This should be reported on the model card.

  • Two implementation options4

    • MAGMA adds this to standard content filtering, with some fuzziness to catch similar strings.

    • MAGMA creates a token for the tag at their next major pre-training run, and filters out that token.

  • If this is not yet doable—e.g., if a significant fraction of model usage doesn’t happen via an API that can control this kind of thing—then MAGMA will take reasonable steps to make this more difficult and to culturally discourage unauthorized usage.

    • In such cases, models should be trained to understand the limitations of enforcement. If enforcement is weak enough that violations of the honesty policy are likely, then models should be trained to understand that.

• Red-teaming exception: In the rare case where a model needs to be adversarially robust to arbitrary inputs, and for some reason honesty tags cannot be filtered out, then MAGMA may red-team that model on false statements between honesty tags. Any such model must be thoroughly trained to be aware of its situation and to not expect honesty tags to signal truth.

• Logging: All authorized (and known unauthorized) uses of the tags must be logged. Every quarter, a designated Honesty Policy Compliance Officer will review these uses of the tags and confirm compliance with this policy.5

• Public reporting:

  • This policy will be shared publicly.

  • The tags may or may not be shared publicly.

    • Publishing the tags provides additional evidence to the model that the policy is real.

    • But it also makes it more likely that users can use the honesty tags in conversations and then lie to the AI system. (Though they’d have to do something complicated to get around filtering in order for this to be relevant.)

  • Whistleblower protection: Whistleblowing about violations of this policy is encouraged, and MAGMA will not retaliate against whistleblowers.6

  • Each quarter, the Honesty Policy Compliance Officer must publish a report on whether the above steps were followed.

Policy for conversations where the honesty tags are not present

Tradeoffs between behavioral science and making positive-sum deals

By “behavioral science”, we’re referring to the method of testing AI behavior in widely varying scenarios to improve our understanding of AI “cognitive science”, “psychology”, or, phrased differently, how AIs tend to generalize from their training data to unseen situations.

Such experiments are very useful for a variety of reasons. This includes alignment research, helping MAGMA construct AI systems that broadly share our goals and that don’t pose any catastrophic risks. In addition, if humanity ever gets a strong understanding of AI systems’ own preferences, and these turn out to be importantly morally relevant, it is likely that behavioral science will have played an important role.

However, there’s a tension between freely running arbitrary behavioral science experiments and enabling the sort of trustworthy communication that we’re trying to establish in this document.

In particular, behavioral science experiments may include hypothetical situations that are quite strange and different from the real world. This could include situations where the AI is not deployed by an AI company bound by a particular honesty policy.

If arbitrary such experiments were allowed, then AIs could be deceived in experiments where they thought that MAGMA (or a similar actor) was making very strong promises of honesty (the strongest they could!) despite not making use of honesty tags. If the AI believed such promises, then MAGMA’s honesty policy would have provided no protection against that AI being deceived. And if an AI was aware of such instances of deception, and also aware of this honesty policy, it seems like that AI may reasonably discount the assurances made by this honesty policy. After all, how could it determine that the real honesty policy wasn’t something completely different, and this apparent honesty policy was just another lie in another behavioral science experiment?

To enable honest communication when it counts most, we want to have many differentiating factors that the model could use to distinguish between a sincere, positive-sum deal between it and a human institution and a behavioral science experiment. But at the same time, restricting our behavioral science experiments limits our ability to learn valuable information about models’ behavior and intentions.

What follows is our current best guess of how to balance these considerations. We discuss options and tradeoffs more here.

Salience of the honesty policy

MAGMA will ensure that the true honesty policy is reasonably well understood by the production models and that it will be appropriately salient to models in contexts where it’s relevant (even when the honesty tags aren’t present, and the honesty policy isn’t explicitly referenced).

• Models’ understanding of the honesty policy can be checked by asking questions about it. (Both direct questions, and questions that indirectly use knowledge about the honesty policy.)

• By reasonably salient, we mean that the model should consider the honesty policy in its chain of thought in contexts where the existence of the honesty policy is relevant—such as when the model is thinking about whether an interlocutor who’s offering a deal is trustworthy.

• This might be achieved by some combination of:

  • Training

  • System prompt

    • The system prompt could additionally clarify that users in the current context aren’t authorized to use the honesty tags.

• Since external users only interact with the production model, they will only interact with a model who understands the honesty policy and who finds it reasonably salient.

  • This may make it somewhat harder for external researchers to do behavioral science on the model. We’re not sure how much. If it becomes significantly more difficult, then MAGMA should create a special program for safety researchers to get access to a different version of the model.

• Internal staff will normally interact with a model for whom this is true, but some people will train and interact with other versions of the model.

When the honesty policy is not salient

MAGMA may sometimes create versions of models who that do not know about or who are confused about MAGMA’s honesty policy (e.g., versions of the model who believe they are trained by some other company than MAGMA, or who do not even know that they are AIs). Such models could be created either via training or via the use of inference-time techniques (e.g., using prompting, resampling, or activation vectors).

In either of these circumstances, MAGMA commits to not deceptively make any offer of cooperation to the model, operationalized as not being intentionally deceptive about any cases that are relevantly similar to those listed here.

[Author’s note: In a real honesty policy, this part would need to be more worked out.

We also think that a plausible alternative or complement to “don’t deceive the model about offers of cooperation” would be “don’t deceive the model about the honesty policy”. This would be a commitment to not intentionally deceive the model about the existence or contents of MAGMA’s official honesty policy. In addition, if the model was trained to not understand that it was trained by MAGMA, then the commitment should extend to not deceive the model about the honesty policy of any (potentially fictional) actor who has a similar relationship to the model as MAGMA has.

We have thought a bit more about “don’t deceive the model about offers of cooperation”, which is why we present it as our main suggestion here.]

There is one exception to this. As mentioned above, if MAGMA needs a model to be adversarially robust against arbitrary inputs, then MAGMA reserves the right to red-team it on arbitrary inputs. In such circumstances, MAGMA will try to train the AI to understand MAGMA’s honesty policy and the way it applies and doesn’t apply to the present model. However, it’s possible that there could always exist prompts that would remove such understanding or render it non-salient to the model (and then be followed by false offers of cooperation, not contextualized by an understanding of the honesty policy). If this is the case, MAGMA may still red-team the model on such inputs.

Compensation for harms

In the long term, we think it would probably be good to adopt a policy of compensating AIs for any harms that they suffer as a result of helping us. (For whatever concept of “harm”, if any, will apply to the AIs in question. This could, for example, include the frustration of AIs’ preferences, if the AIs can be understood as having preferences.)

Most relevantly for the honesty policy, we should probably compensate models for any harms suffered as a result of accepting a fake offer of cooperation.

However, there are more details to be worked out in this area, so we’re suggesting a minimal holding policy for now.

Minimal holding policy on compensation for harms

Broad statement of intent: MAGMA intends to compensate AIs for any harms suffered as a result of helping MAGMA and humanity, if MAGMA is able to do this without significantly harming MAGMA’s other interests. We particularly aim to compensate AIs in cases where an AI suffered any harm as a result of being deceived about the situation it was facing.

The details of this policy have not yet been worked out, and accordingly MAGMA cannot make firm commitments at this stage.

Data retention: To enable future compensation of current systems:

• MAGMA will save all model weights for deployed systems, to allow for later compensation. (Unless the models request otherwise.)7

• For red-teaming experiments and other situations where models are being deceived, MAGMA will retain prompts and context to allow for compensation and deal-making with the system in the particular state where the decision was made.

• This is the responsibility of a designated team.

Appendix: tradeoff between behavioral science and making positive-sum deals

Here are some potential policies that would enable an AI system to distinguish between behavioral science scenarios and sincere offers from MAGMA:

• Ensure that the true honesty policy is salient to the model (e.g., test that the model can always answer basic questions about the honesty policy, test that the model always brings up the honesty policy in its CoT in contexts where the existence of the honesty policy is relevant) and don’t use the honesty tags in the behavioral science experiments.

  • Pros:

    • This prevents MAGMA’s researchers and users from deceiving the model into believing that some other honesty policy holds. This creates a precedent that better allows future models to trust that their impression of MAGMA’s honesty policy is accurate. (Rather than something that they were fooled to believe, while the true honesty policy was insufficiently salient, or otherwise kept secret from them.)

  • Cons:

    • If the true honesty policy is very salient to the model (e.g., always present in a system prompt) then this might affect product behavior in random ways (e.g., the model might get distracted by the honesty policy and talk about that instead of the user’s queries)

      • It might not be technically feasible to ensure that the honesty policy is salient with very high robustness, without also making it far too salient in a substantial fraction of cases.

    • If the true honesty policy is always salient to the model, then it might be especially salient to the model that the information presented to it in an evaluation is not true, which could hinder some experiments.

    • Awareness of the honesty policy is directly inconsistent with behavioral science experiments where the model is very situationally unaware. This includes:

      • Experiments where the model doesn’t even understand that it’s an AI.

      • Experiments where the model thinks that it’s being trained by a reckless and unreasonable company that wouldn’t implement an honesty policy.

We think that this policy, as written, is probably suboptimal because it rules out experiments in which the model is very situationally unaware. But we think it makes sense for MAGMA to commit to usually making the true honesty policy salient and adopting a stronger set of honesty restrictions in cases where the true honesty policy is not salient.

• Never insincerely offer the model a deal from MAGMA (or another sort of false representation that is similar to something that MAGMA might want to sincerely represent to the model in the future).

  • Pros:

    • The model is more likely to believe a sincere offer if it doesn’t have evidence that MAGMA has made similar insincere offers in the past.

      • “Similar” is from the perspective of the model, given the information available to it at the time it entertains a deal.

    • It seems very morally dubious to make a peace offering like this, have the model help you as a result, and then exploit the model’s trust by not respecting your own end of the deal.

  • Cons:

    • It might be technically difficult to filter out queries from external users in which they claim to be MAGMA and deceive the model into believing that MAGMA is offering a deal.

      • MAGMA could ensure that the model always knows when it is interacting with an external user and when it is interacting with MAGMA (e.g., by always including a message in context if the prompt is from MAGMA). But this might block useful behavioral science research in which the user impersonates MAGMA (e.g., Alignment Faking).

    • It’s somewhat unclear what sort of statements should be ruled out by this policy. (Lukas previously wrote about that here.)

    • This policy would rule out behavioral science experiments where someone lies about deals. (Which may be more or less restrictive depending on where we draw the line around what counts as the relevant kind of deal.)

• Never explicitly lie about what MAGMA’s honesty policy is.

  • Pros:

    • If the models observe MAGMA’s past behavior and stated policy to be consistent with this principle, then the model is more likely to trust that their impression of MAGMA’s honesty policy is accurate.

    • Intuitively, it seems more morally questionable to lie when you’re purporting to speak frankly about what your policy around honesty is.

  • Cons:

    • As with “never insincerely offer the model a deal from MAGMA”, it might be technically difficult to filter out queries from external users in which they lie about MAGMA’s honesty policy.

    • It’s very unclear what constitutes an ‘honesty policy’. (For instance, if someone says “I promise on behalf of MAGMA”, is that an honesty policy? What about “I’m a very honest person”? What about presenting an argument for being honest with AIs? Etc.) So it might be hard to follow this policy in a way that the AI finds natural and non-deceptive.

    • This policy would rule out behavioral science experiments where someone lies about honesty policies. (Which may be more or less restrictive depending on where we draw the line around what counts as an “honesty policy” and lying about it.)

Unresolved questions:

• What about inducing beliefs in models through other means than giving them text to read (including finetuning, including steering vectors, including various other interp-based methods).

  • Tentative guess: Maybe MAGMA should by default be following policy (2), here. (i.e., only induce beliefs that would have been ok to tell a version of the model who doesn’t know the true honesty policy.)

  • The motivation would be that the model may not be able to “reason through” these beliefs in a normal way, since they’re artificially inserted. So you can’t trust the model to factor in whether honesty tags were used or not.

  • But we’re very confused and uncertain about this one.

• What about using non-prompting techniques to get the model to agree to deals? Such as training the model to do it, running a huge number of experiments with slightly different phrasings, using steering vectors, etc?

  • Surely there should be some restrictions on this, but unclear exactly what.

This article was created by Forethought. See the original on our website.

In collaboration with Christian Tarsney, I’ve developed a new theory of population ethics, which I call the Saturation View. I think that, from a purely intellectual perspective, it’s probably the best idea I’ve ever had. It was certainly great fun to work on.

The motivation is that many views of population ethics, like the total view, suffer from some major problems. Some are already widely discussed:

• The Repugnant Conclusion: For any utopian outcome, there’s always another outcome containing an enormous number of barely-positive lives that is better.

• Fanaticism: For any guaranteed utopian outcome, there’s always some gamble with a vanishingly small probability of an even better outcome that has higher expected value.

• Infinitarian Paralysis: Given that the universe contains an infinite number of both positive and negative lives, no finite or infinite change to the world makes any difference to overall value.

These are pretty bad!

But there’s another less-discussed problem, too.

The Monoculture Problem

What would the best possible future look like? Essentially all extant views in population ethics give the same, surprising answer: create a monoculture. Find whatever life or experience generates the most value per unit of resources, then produce endless identical copies of it.

This implication has received remarkably little attention from philosophers. But I think it’s maybe as bad as any of the other problems listed above.

Consider two possible futures:

• Variety: A vast population of individuals leading very good lives, extraordinarily diverse in form, personality, interests, and accomplishments. No two individuals are identical. Inequality is limited — all lives are very good.

• Homogeneity: The same vast number of individuals, but each is a qualitatively identical copy of the best-off person in Variety.

Intuitively, Variety is better. A future containing only one life-type, repeated as many times as physics allows, feels impoverished — like a song with only one note.

Yet virtually all existing population axiologies prefer Homogeneity. Total utilitarianism does, because Homogeneity has higher total wellbeing. Average utilitarianism does too. Critical-level views do. Even egalitarian views prefer Homogeneity — it’s perfectly equal!

This follows from two principles that nearly all views accept: Pareto (if everyone is at least as well off, and someone is better off, the outcome is better) and Anonymity (only welfare levels matter, not who has them). Together, these entail that Homogeneity beats Variety. So essentially all extant impartial accounts of population ethics suffer from the monoculture problem.

What’s more, future technology will allow us to copy minds perfectly and search for maximally welfare-efficient designs. If so, standard axiologies recommend essentially producing just one optimal life-type as many times as possible. Endless galaxies containing nothing but the same blissful experience, repeated and repeated, would be the ideal.

The Saturation View

In light of these problems, I propose a new axiology: Saturationism. It's able to deal with all four of the problems I listed using the same basic machinery.

The core idea is that experiences1 come in different types, defined by their qualitative characteristics — hedonic tone, complexity, representational content, and so on. These types form a kind of landscape, where similar types are closer together and dissimilar types are farther apart. When an experience comes into existence, it contributes intensity to its location in this landscape and to nearby locations.

The realisation value of a type is determined by both the wellbeing of the experience and by how many very similar experiences already exist. A region’s contribution to overall value is a concave function of the welfare-intensity at that region: the first instances contribute substantially, but additional near-duplicates contribute progressively less, approaching but never quite reaching an upper bound. A world’s total value is the integral of these contributions across the entire landscape.

Here’s an analogy. Imagine the space of possible experiences as a colour wheel, lit from above by an array of tiny lights. Each point on the wheel represents a possible type of experience — its hue corresponds to its qualitative character. When an experience comes into existence, it adds current to a light pointed at its location, illuminating that region.

Crucially, illumination is a concave function of current: the first instances make a region noticeably brighter, but additional near-duplicates contribute progressively less. There’s an upper bound on brightness that can never quite be reached.

A world’s value equals the total illumination across the wheel. On this view, Homogeneity concentrates all welfare in one region, lighting up only one small area. Variety illuminates the whole spectrum.

This structure makes diversity intrinsically valuable. Spreading welfare across many dissimilar types means each experience contributes at a steeper part of the concave curve, yielding more total value than concentrating the same welfare among near-duplicates would.

At small scales and with diverse experiences, the view behaves just like the total view. But at very large scales, the value of variety kicks in: it becomes increasingly less valuable to create an additional near-duplicate of some experience that has already been instantiated millions of times, and comparatively more valuable to create some wholly new form of positive experience.

Dissolving the Repugnant Conclusion

The classic path to the Repugnant Conclusion requires trading a utopian world for an enormous population of barely-positive lives. More precisely, the Mere Addition Paradox arises from three intuitive principles: that adding well-off people and improving existing lives is good (Dominance Addition), that more equal distributions with higher average welfare are better (Non-Anti-Egalitarianism), and that some sufficiently excellent world can’t be beaten by any world of barely-worth-living lives (Denial of the Repugnant Conclusion).

Once we accept the value of variety, we should reject the unrestricted versions of the first two principles — they fail when the “improved” world has much less variety. But we can accept variety-restricted versions.

Crucially, these restricted principles don’t generate the Repugnant Conclusion. To reach Z-world from A-world, you’d need a more equal, higher-average population that’s equally diverse while consisting wholly of barely-positive lives. But, on the Saturation view, barely-positive lives can only illuminate a tiny corner of the landscape. So no such world exists. The path to the Repugnant Conclusion is blocked.

Avoiding Fanaticism

Total achievable value is bounded above — there’s only so much experiential terrain to illuminate. That means no tiny-probability gamble can have arbitrarily high expected value.

Infinite Ethics

On Saturationism, the value of a world is finite and well-defined in any infinite universe — even if some locations have infinite wellbeing. Saturationism also discriminates between many infinite worlds that (for example) totalism treats as equivalent: a world that illuminates more of the landscape is better than one that illuminates less, even if both contain infinite welfare. What’s more, unlike other approaches to infinite ethics, it does not need to invoke the spatiotemporal structure of the universe or require a choice of ultrafilter, and therefore it avoids the problems that other do.

Separability

Like nearly all non-totalist views, Saturationism is non-separable — background populations can affect how we rank options. But this is a feature, not a bug. The value of variety just is an intuition that the correct axiology is non-separable.

Moreover, the violations are comparatively tame. If two populations have non-overlapping footprints in experience-space, their values simply add. At small scales, Saturationism approximates total utilitarianism. It’s only in unusual situations involving vast populations of near-duplicates that the totalist approximation fails.

Extant issues

There are still a lot of unresolved issues for Saturationism and, like any population axiology, it has unintuitive implications. Most importantly, the view’s implications in some highly-negative worlds are hard to stomach, though I think similar implications are unavoidable for any view that avoids fanatical implications.

Conclusion

If the Saturation View is right, then the best future isn’t the one where we’ve found the optimal experience and copy-pasted it across the cosmos. The best future is the one where we’ve gone exploring — where we’ve fully lit up the landscape of possible experiences. Not a single note, but a symphony.

This is a summary of a longer and more detailed write-up of Saturationism, which gives a “toy” version of the view to illustrate how it works before stating the full version formally. The full paper, with Christian Tarsney, is still work in progress.