Outline

1. In the introduction, I explain what I’ll be arguing in the piece—namely, that 1) it is very important that we hand off most high-stakes decisions to AI; and 2) the kinds of AIs we should hand off to are philosophically reflective AIs with values that shift over time as a result of reflection. We should ensure the AIs that we build explicitly consider moral arguments, and sometimes change their priorities on the basis of moral argumentation. This would be much better than locking in current values, retaining human control, or allowing a future dominated by whatever haphazard mix of humans and AIs emerges naturally.

2. Why hand off? In this section, I give the main arguments in favor of the thesis. In short, I argue we should hand off because: 1) AIs are likely to be more virtuous than people; 2) AIs are likely to be much smarter and better at making decisions than people; and 3) AIs are likely to be better at quickly navigating the difficult decisions that one must make during an intelligence explosion. I argue we should hand off to philosophically reflective AIs because 1) reflective AIs are likelier to get the right answers to important moral questions, or as close to the right answers as one can get, than the default scenario, which raises the odds of a near-best world; 2) reflective AIs are less likely to make horrendous moral errors, leading to a wide-scale moral catastrophe, both than humans and non-reflective AIs.

3. Would handoff disenfranchise humans? Here I address the worry that handoff would disenfranchise humans by taking decision-making out of our hands. My reply is that: 1) the default trajectory without handoff disenfranchises far more expected beings in more serious ways (odds are non-trivial of digital minds being seriously disenfranchised)—in fact, one of the most promising proposals for how to hand off is simply to give basic political rights to AIs; 2) a compromise solution where humans retain nearby resources can allow humans to be in the loop on the decisions we care about most; 3) on a number of plausible views, the determinant of the desirability of some system of decision-making is the quality of the decisions, rather than whether there’s democratic input. Given how enormous the stakes could be—affecting billions of times more sentient beings than there are humans—it’s hard to think the harms of human disenfranchisement are so great as to make handoff undesirable.

4. Would we like their advice? In this section, I address concerns that if AIs pursue the good, the end result might be alien and divorced from human values. In response I argue: 1) the future, by default, is likely to be highly suboptimal in many respects, so for handoff to be desirable, it must only beat the alternative; 2) values in the future are likely to be very different from current values, because they’ll shift dramatically over long time scales, so this is not a unique downside; 3) one could reach a deal where current values govern the surrounding region of space, while distant places in space are geared towards the production of maximal value—this would be desirable from the perspectives of both common-sense and cosmic ethics; 4) on many views in philosophy, the stuff that’s objectively valuable is what we’d want if we were ideally reflective—but if you know you’d be motivated to bring something about if you were wiser and more reflective, then that gives you a reason to bring it about; 5) only a relatively narrow subset of views hold that there are objective values but don’t require pursuing them. If there is a conflict between what we want and what’s objectively valuable, on standard views, we should simply go with what’s objectively valuable. And if there’s no objective value, then the dilemma doesn’t arise at all—and philosophically reflective AIs will simply produce an upgraded version of human values, rather than discover some far-flung and potentially alien truths.

5. Worries about handoff. In this section, I address concerns about how handoff might be implemented involving alignment, whether AI would be sufficiently ethically reflective, whether handoff would enable power grabs, whether it would cause lock-in, and whether it would be worse than some hybrid system.

6. The conclusion recaps the main points of the piece.

1 Introduction

“How horrible!”

“Perhaps how wonderful! Think, that for all time, all conflicts are finally evitable. Only the Machines, from now on, are inevitable!”

—Isaac Asimov, “The Evitable Conflict”

Is the optimal future one in which we hand off important moral decisions to AI? Should, in other words, AIs be the ones making most high-stakes decisions instead of us? And if so, what kinds of AIs should we hand off to?

Many people envision a handoff scenario as a terrifying and potentially existential catastrophe. They worry about humans being locked out of the levers of power and having our share of resources slowly dwindle as AIs seize control of more and more institutions. I worry somewhat about this kind of scenario. But in my view, we should be worried primarily about the wrong kind of handoff occurring, not about handoff writ large. The future we should aim for is a kind of handoff. This piece lays out that perspective.

There are different ways handoff could work. We could hand off to AIs that roughly mirror human values—perhaps slightly changing them to remove inconsistencies. Alternatively, we could hand off to the kinds of AIs that deeply and carefully philosophize—figuring out what’s best to do and doing that, even if it diverges substantially from current practice. This piece argues that the second kind of handoff is very important for avoiding serious moral error. I am very worried about the possibility of AIs locking in the moral beliefs of 21st-century humans.1

I am also worried about scenarios where amoral profit-maximizing AIs without any clear moral aims take control of most of the world’s resources.

Note: these core claims are dissociable. You could think we should hand off to AI, but we shouldn’t hand off to reflective AIs that update their judgments in response to philosophizing. Alternatively, you could think that handing off would be a bad thing, but that if we are going to hand off, we ought to hand off to reflective AIs.

In this piece, section 2 will present the main argument for handoff—that we should expect AIs to make much better decisions than us on a range of consequential subjects. It will also discuss the case for handing off to reflective AIs that are willing to update their values in response to careful philosophizing, rather than locking in some version of current human values, arguing that a world where AI locks in something in the vicinity of current values likely misses out on almost all possible value. Section 3 will discuss whether handoff would be bad because it disenfranchises humans or gradually disempowers them. Section 4 will discuss the concern that AIs will discover the moral truths, but those truths will be strange and alien, so this will be bad by the lights of current human values. Section 5 will discuss some more granular worries about handoff. Section 6 will conclude.

This piece is primarily about whether to hand off and not when to hand off, though the considerations I present should make one somewhat worried about short-term actions to prevent handoff, because such actions lower the odds that handoff ever happens. The considerations I present, if correct, also give some reason for wariness about many actions to reduce the odds of gradual disempowerment scenarios.2

My claim is that the best future for humanity involves us being disempowered in some sense, in that humans aren’t making most high-stakes decisions. As an analogy, representative democracy is, in some sense, a form of handoff—we hand off power to our elected representatives. Handing off power to wise AIs could be even better.

This has a number of important practical implications. It means that accelerating AI macrostrategy is especially important, so that at the time critical decisions are being made, wise and philosophically reflective AIs are in the loop. It similarly provides reason to support work on making AI have virtuous character, rather than just follow rules. Model constitutions should express commitment to following the true moral theory, insofar as there is one, and if not, following some reasonable compromise across moral theories. Anthropic’s language here seems good.3

What kind of handoff scenarios should we aim for? One shouldn’t be too specific about these sorts of things. The future is hard to forecast and rarely follows simple models. But I’ll describe the handoff scenarios that seem most desirable, and what traits in AI we should look for before handing off critical decisions to them.

There are different ways handoff could go well. One way resembles, in certain respects, the gradual disempowerment scenario (the core difference being that this would hand off to morally scrupulous AIs rather than myopic profit-maximizers). AI will make increasingly large numbers of critical decisions, because of its cognitive superiority. By the end, nearly every important decision will be made by wise AIs, who will hopefully, by that time, have been granted political rights. As an analogy, future generations eventually gain control of most of societal decision-making—yet this isn’t because there’s ever some deliberate choice to hand off power to the next generation. It occurs naturally with time.

A number of people seem to conceive of handoff as a strange abrogation of the liberal order—one that replaces human decision-making with AI. But this doesn’t have to be. One of the more promising ways of handing off would be giving economic and political rights to digital minds. Because digital minds could be so numerous, eventually this would lead to them making nearly all decisions. This would, in fact, be squarely in accordance with the norms of the liberal tradition, for it would give rights to morally important welfare subjects. There are other ways a good kind of handoff could occur involving dealmaking. Different actors might each think they’re morally right, and thus agree to a deal where AIs are allowed to dictate the future—each party thinking that doing so would favor their priorities. Alternatively, if AIs improve collective decision-making, people might intuitively come to appreciate the weight of human moral error and thus permit AIs to make the highest-stakes decisions.

Before handing off most critical decisions to AI, we should look for each of the following:

• Alignment: We should have strong evidence that AIs don’t have underlying scheming motivations. This could take the form of consistent friendly behavior even in important situations where they have the option to misbehave, or it could take the form of high-octane interpretability work that lets us ascertain their motivations.

• Philosophical aptitude: AIs should be genuinely interested in finding the moral truths. They should sometimes hold moral views that people don’t hold, and be willing to change their mind in response to new evidence. One could survey professional philosophers to see if they consider AIs better than the best humans at philosophy and could design philosophy benchmarks to test this.

• No lock-in: Before handing off to AI, we should ensure that the AIs are willing to change their values over time. Their values should change in response to new evidence and they should not be interested in locking in whatever it is that they happen to currently value (absent some strong reason to think they stumbled across the correct set of values).

• Coherence: Current AIs don’t have consistent and stable preferences across time. We should only hand off after AIs display these kinds of preferences. This doesn’t mean they never change their minds, but it does mean that they have relatively consistent desires that only change in response to good reasons. For comparison, humans often change our minds, but have far more rooted preferences than LLMs of today.

• Intelligence: AIs should display the level of intelligence needed to make the decisions that we put in their hands. When AIs are only a bit more intelligent than us, they can plausibly make some important decisions. Only after they display immense cognitive superiority should we turn over most decisions to them.

• Tested: We should only hand off big-picture planning to AI after it’s been able to make good low-stakes decisions (say, the running of a company). Before all decisions are handed off, there should be some critical period where high-stakes decisions are made mostly in consultation with AI.

Now, you might wonder: if handoff occurs in a way that’s gradual and decentralized, how do we ensure that these conditions are met? My guess, however, is that even if handoff is a slow and gradual process, there will be times when discrete decisions need to be made. For example, we might imagine AI growing more agent-like, beginning to perform a healthy share of economically viable tasks, contributing to cultural and social life, and behaving in ways resembling a conscious agent. This alone wouldn’t produce handoff. To hand off, we’d need to eventually give AIs control over the legal system. Thus, even in gradual handoff scenarios, there will be specific actions that need to be taken to facilitate handoff.

Alternatively, we could take actions ahead of time that would shift the kind of handoff that would occur. Private AI companies or governments should ensure the AI being created possesses virtues and a desire for philosophical reflection. That way, when handoff occurs, it will be to morally reflective AIs.

2 Why hand off?

2.1 Why hand off at all?

The main reason to hand off to AI is that AI could be much better at making decisions than people in three key respects: virtue, intellectual capability, and speed.

First, virtue: humans possess each of the virtues only to a fairly limited degree. Yet in principle, AIs could have arbitrarily great degrees of any virtue. Because they are built with moral directives in mind, rather than by a blind and morally indifferent evolutionary process, there isn’t as much of a limit to how morally scrupulous, compassionate, honorable, and so on we could make them. This means that if we hand off correctly, it is reasonably likely that we’d have supremely wise and virtuous decision-makers.

AIs are already nicer, friendlier, and more reflective than people, and this is only likely to improve over time.4

If you ask AI models about high-stakes questions, you will generally get far more reasonable answers than you’d get from most people. Crucially, AI models are in their early stages—we should expect them to get better over time.

While Claude is not sufficiently coherent to be president, if it was, I suspect I would generally prefer the decisions of Claude to most presidents. Same with the other AI models (at least, insofar as one removed the sanitization that prohibits them from giving real opinions). And while models currently struggle to accomplish tasks over long time horizons, hallucinate, and so on, given the extremely rapid rates of progress, it would be surprising if these trends persist indefinitely.

Second, intellectual competence: we should expect a world of superintelligence to require making a number of very difficult decisions. Superintelligence could enable AIs to correctly make important decisions that depend on being right on non-moral matters, where the answers aren’t obvious. Some of the challenges in the future include:

• Divvying up space resources in a way that lets goodness compete. There are plausible future scenarios where competition will squander the cosmic commons, so that resources will be spent competing rather than bringing about value.

• Mitigating existential threats, including intergalactic ones. Future technology could enable small-scale groups to threaten huge intergalactic civilizations.

• Dealing with the risks posed by a world of superintelligence.

Third, speed: in a world of very rapid technological progress, we’ll have to make a large number of these decisions extremely quickly. It isn’t at all obvious that humans can make these decisions well, in a way that prevents civilization from being irreparably ruined. As AIs get increasingly complex, the difficulty of decisions needed to manage them will also get very complex. To mitigate some threat, decision-making might have to occur more quickly than the fastest human decision-making.

The case for handoff is thus relatively straightforward: in the limit, AIs will be much better than humans and better equipped to navigate a complex and rapidly shifting future. To reduce the risk of colossal mistakes, then, it’s important that humans aren’t in control, but instead the already friendly and soon to be superintelligent beings are.

2.2 Why hand off to reflective AIs?

2.2.1 How the future might be

There are different AIs that we could hand off to. On the one hand, we could hand off to AIs that judiciously reflect and try to pursue the good, whatever it looks like. On the other hand, we could hand off to AIs that pursue some mild variant of human values. In this section, I’ll explain why I favor the first. Consider the following taxonomy:

1. Optimal world: the world is optimized according to the right set of values.5

2. Compromise world: the world is optimized according to a compromise among reasonable moral values.

3. Unguided world: the world is not optimized according to any specific set of moral values. Instead, it bears more resemblance to the current world, where decision-making isn’t optimal by the lights either of the true moral theory or any compromise among the leading moral theories.

My guess is handoff to reflective and superintelligent AIs done correctly probably gets 1 if moral realism is true and 2 if it isn’t. If we don’t hand off, my guess is we get 3. Later pieces will discuss in more detail the odds of getting a near-optimal world and the prospects for AI making philosophical progress. My guess is scenario 2 has below 10% the value of scenario 1 and scenario 3 has below 10% the value of scenario 2, for reasons I will lay out.

2.2.2 Optimal worlds contain a big slice of future value

Better Futures makes the case that a pretty big slice of expected future value is contained in the narrow slice of worlds that are close to the best. There are a number of high-stakes moral questions which we have to answer correctly to not lose out on almost all future value. It’s not at all obvious what the answers to these are. For example, two of the most plausible views of population ethics are totalism (which says the welfare value of a population is purely a function of total welfare) and critical level theories, which hold that adding an extra happy life is good only so long as their welfare surpasses a particular level. By the lights of totalism, the ideal world according to critical level theories might have value on the order of 1% of what it could be (if the best way to maximize utility is to proliferate low-welfare lives). By the lights of critical level theories, the optimal world according to totalism might be actively bad—so long as it’s stocked with people below the critical level.

So in short, nearly all value is lost unless we get the right answer to a bunch of very difficult ethical questions that philosophers who spend their lives working on haven’t agreed on the answer to. It seems unlikely that people will solve these on their own. If AIs tell them what the answers are, and these answers diverge from people’s explicit beliefs, people might not believe the AIs (just as people generally don’t take very seriously expert testimony on non-empirical matters). Similarly, people spend relatively little time thinking about how they can do the most good with, for instance, their career. If humans received testimony about what ought to be done that diverged from what most people favored, my guess is that they generally wouldn’t care about the answers. If humans remain in control and learn that they ought to create the repugnant conclusion world, probably they wouldn’t do so.

It’s less obvious that AIs wouldn’t converge on the right answers. I’ll discuss in a later piece proposals for getting AI to get the right answers to philosophical questions, as well as reasons to think that they are reasonably likely to get things right. Given how unlikely it is that humans will get the right answers to the moral questions, insofar as there are right answers, probably the prospects for AI are better.

2.2.3 Compromise worlds>>unguided worlds

If there aren’t moral facts, then AIs would still be able to work out some optimal arrangement that is great according to all sets of reasonable values. If there aren’t moral facts, then ideally the AI should make decisions according to the verdicts of a parliament of the theories that ideally reflective humans would reach. So suppose that after reflecting, 50% of humans would end up totalists, 30% would adopt some version of the person-affecting view, and 20% would adopt critical level theories. The AI would then make decisions as if there was a parliament comprised of 50% totalists, 30% person-affecting view adoptees, and 20% critical level theorists.

It is unclear exactly how good a compromise across different theories ends up by the lights of each particular theory, but likely far better than the human-run default. In other words, 2 (compromise world) is much better than 3 (unguided world). Here is why.

In the future, given advanced technology, very large amounts of value should be realizable. But if future resources aren’t directed specifically towards the production of value by most agents, most resources are likely to be used in a highly suboptimal way, and most value is likely to be lost. Moral errors become a much bigger deal in a world of much greater technological competence.

My guess is that the default human-controlled scenarios do not involve humans thinking very hard about what to do and doing anything like what is optimal. Certainly humans have so far not spent much time on this task—consulting with philosophers on what is optimal and so on. This sort of thing will become easier in a world of advanced AI, but it’s already pretty easy; if virtually no one does it, and if people often continue performing actions even after they believe them to be wrong (more on this in later pieces), then we should be pessimistic that this will change dramatically in the future.

The enormousness of the gulf between scenarios 2 and 3 becomes clearer when one thinks vividly about what the compromise world would look like. Perhaps it would involve using space resources to create maximally large numbers of happy digital minds.6

These minds would be supremely well-off across all theories of well-being. But it is hard to imagine in an unguided world space resources being used optimally to create very large numbers of well-off minds, just as in the world today, there has been no systematic effort to use resources in ways that are optimal across a range of moral theories. This conclusion is bolstered by considerations I’ll provide in later pieces, that people generally don’t have much moral motivation, and don’t care very much about doing good things that aren’t personally resonant.

2.2.4 Moral errors

Humans have a long history of making serious moral errors. As Evan Williams writes, “Show me one society, other than our own, that did not engage in systematic and oppressive discrimination on the basis of race, gender, religion, parentage, or other irrelevancy, that did not launch unnecessary wars or generally treat foreigners as a resource to be mercilessly exploited, and that did not sanction the torturing of criminals, witnesses, and/or POWs as a matter of course. I doubt that there is even one; certainly there are not many.” It would be very suspicious if we were the first society in history that did not go massively morally wrong. And while AI advice can help mitigate this to some degree, it’s far from obvious that it would be sufficient to eliminate moral errors.

There are a number of respects in which it is very plausible that we go morally wrong. To take one example of a judgment that many philosophers think is in error, consider wild animal suffering. Almost every sentient being is a wild animal. They suffer and experience joy in truly gargantuan quantities. Yet they are counted for nothing in most decision-making, despite there being strong arguments for considering their interests. Crucially, this is not because people have in general spent a lot of time thinking about wild animal suffering and concluded that it doesn’t matter. It’s that most people haven’t thought about it at all. Many other examples could be given, depending on one’s moral views. If superintelligent AI told people that wild animal suffering was a big deal, probably most people wouldn’t care much.

In the future, nearly every expected sentient being will be digital. Nearly all expected future welfare will be experienced by digital minds. This follows even if you have a low credence in the possibility of digital sentience because if digital minds are possible, they could be produced in enormous numbers. It is easy to imagine a scenario where humans do not take seriously the interests of at least some digital minds, and horrific suffering is doled out at cosmic scales. Certainly it would not be the first time humans have neglected the welfare of those different from themselves. You do not have to be a consequentialist to think the possibility of neglecting the interests of galaxies full of conscious and intelligent beings is a terrifying one.

2.2.5 Handoff mitigates odds of moral error

Handing off important decisions lowers the probability of catastrophic moral errors. This both increases the odds of achieving optimal and compromise worlds and lowers the odds of making catastrophic moral errors.

The first way handoff lowers the odds of moral error is by having decisions be made on explicitly moral grounds. If we hand off decisions to supremely virtuous AIs trying to act morally, then we won’t sleepwalk into doing obviously evil things. Decisions will be made consciously optimizing for doing what is right, instead of whatever suboptimal arrangements make it through consensus-making mechanisms. Thinking about morality before making choices doesn’t guarantee that we’ll always do the right thing, but it does lower the odds that we do things that can only be done by explicitly neglecting moral considerations. This is especially plausible if the decision-makers are virtuous.

Now you might wonder: why would people ever hand off to AIs if the AIs disagree with them about morality? But this is, in essence, very similar to a kind of handoff that people do support: handing off the future to future generations. Even if people have some disagreements with the values of future generations, they’d generally oppose a process for locking in current values, and support the process of open-ended reflection that leads to better values over time. A world of advanced AI might be similar. In addition, given AIs’ cognitive superiority, there might be strong incentives in the direction of handing off, just as a CEO might hand off to a successor with more technical competence, even if they share some non-overlapping values.

A second way handoff to reflective AIs mitigates the odds of serious moral error is by ensuring careful reflection. Insofar as AIs are able to carefully philosophize and try to avoid moral error, and they are superintelligent, they’ll be able to avoid doing morally indefensible things. This becomes especially plausible if one buys the previous considerations: that AIs are likely to be very virtuous.

There is one last consideration in favor of handoff to reflective AIs (which I’ll discuss more in section 4). Over long time scales, our values will naturally drift dramatically. The only way to prevent that is to lock in our current values, which would be very bad—just think about any past society locking in its values. Thus, if values changing dramatically over long time scales is inevitable, the future won’t be populated with our current values: the best hope is that it’s populated by either objectively right values or some compromise across reasonable values.

3 Would handoff disenfranchise humans?

Here’s one concern that you might have about handoff: if AIs are the ones making decisions, then this will mean that the substantial majority of humans aren’t in charge of making important decisions. Just as we should oppose a dictator, even if benevolent, arguably we should oppose superintelligent AIs making most important decisions, even if the process of them gaining power was non-coercive.

My guess is that this isn’t a huge downside to the kind of handoff I advocate, where we allow kind and morally reflective AIs to make the majority of future decisions—e.g. by granting them political rights. First of all, if concerns about disenfranchisement are correct, then if we have AIs that are better at moral reasoning than us, they’d likely be aware of this fact. Thus, if the best way to govern the long-term future is to allow pretty laissez-faire distribution of resources without much top-down decision-making, then the AIs would be aware of that fact and allow such a distribution.

My guess is that the total amount of expected disenfranchisement goes down if AIs have more power. As already discussed, almost every expected sentient being is likely to be digital. Insofar as the status quo might disenfranchise almost all future beings in a way far deeper than their simply not being primary determinants of the democratic process, it is hard to see this as a serious downside of handoff, rather than a point in its favor. That this mass neglect of the interests of future digital minds would be bad follows from very modest ethical principles. And the only way to prevent handoff, in a world of very numerous digital minds, is to disenfranchise them.

Second, handoff is compatible with a compromise solution that allows humans to retain significant control. Humans’ preferences, in general, don’t give any especially strong weight towards using any significant share of the universe’s resources. Thus, there could be a compromise handoff solution, whereby humans get unfettered control over the solar system and a big slice of resources, but the remainder of the cosmos’s resources are spent in accordance with the AI’s decisions on hugely important moral pursuits. One point favoring such an arrangement is that it would take a very long time to reach the distant space resources, so those who don’t care very much about what happens in the distant future are likely to care much less about using most of the universe’s resources. Later pieces will discuss this possibility more.

Third, concerns about disenfranchisement are morally controversial. On a number of plausible views, what matters with respect to societal decision-making is how good the decisions are, instead of who is making them. We already elect representatives, instead of deciding directly upon every important decision. We similarly prohibit children from voting, and few think this is an objectionable kind of disenfranchisement. It doesn’t seem obvious that one has an inalienable right to make hugely consequential decisions on matters that they’re barely informed about which affect others in enormous numbers—e.g. we wouldn’t think democratic input from people who know nothing about cancer treatment was morally required before deciding on which cancer treatments to develop. But most voters know relatively little about what they’re voting on. I can’t possibly hope to discuss this literature in detail, so I’ll just state that it’s plausible to me that the value of democratic participation is instrumental.

Even if you think humans being in the loop on high-stakes decisions is important, it’s not clear that it’s important enough to make handoff undesirable. Remember, the gulf between the AI’s decisions and our own might be truly massive! Galaxies full of value—orders of magnitude more joy and welfare than all that has been experienced so far in human history—may be on the line. In light of a gulf this large, it is at least highly non-obvious that AI making most decisions wouldn’t be worth it.

In Superintelligence, Bostrom estimates that there could be a quadrillion times more digital minds than humans (don’t take the number too literally, but it should give you some sense of the scale). Thus if my arguments are correct, putting decision-making in the hands of AI would in expectation majorly benefit at least billions of beings for every human disenfranchised, if we assume AIs are more likely than humans to count the interests of digital minds. Surely, however, stripping away one being’s ability to contribute to the democratic process is worth benefitting billions (if disenfranchising one person would have prevented a global war that would have wiped out an entire continent, it would have been worth it). So then given how colossal the stakes are, they simply outweigh the downsides of handoff.

4 Would we like their advice?

Here’s one concern you might have with the kind of handoff I advocate, where we hand decisions to reflective moral AIs capable of making progress. Perhaps you just don’t care about the surprising moral facts. Perhaps you have particular values that you care about, but you don’t much care about whether those values are objectively right. Insofar as the reflective AIs ascertain that the way we ought to behave isn’t in accordance with what your actual values are, perhaps you have no desire to follow the AIs’ moral advice.

To use the philosopher’s lingo, you might be concerned about the good de re without being concerned about the good de dicto. That is, there might be particular moral projects you care about without caring generally about the good whatever it happens to be. Perhaps, say, an environmentalist cares about environmental preservation but doesn’t much care if other moral aims turn out to be superior to environmental preservation.

I should note one version of this concern that I think slightly misses the mark. You might worry that moral reflection leads in all sorts of strange and alien directions that don’t track the truth, leading to an ultimate set of values that is neither objectively correct nor represents human values in any important way. However, my proposal is not simply “hand things off to an AI after it carries out arbitrary reflection.” That would be potentially disastrous. Instead, my proposal is that we should try to produce maximally philosophically adept AIs and then, after we’re pretty sure that they’re very philosophically adept, hand things off to them—directing them to pursue whatever’s objectively best if there is such a thing, and if not, to pursue some reasonable compromise across human values. If the AIs discover that there are objective moral truths, we ought to follow those truths. If they discover that there aren’t, then we should task them with pursuing some suitably upgraded compromise of human values. The proposal for getting AIs to do good philosophy need not involve arbitrarily large amounts of reflection.

Now you might wonder: how would we know if the thing that AI reflectively endorses is objectively valuable vs just well-regarded by the AI but lacking objective value? The answer is: we ask the AI after we’ve gotten some assurance as to its philosophical aptitude (later pieces will discuss how we can get such assurance). If we have AIs that can figure out what the objective moral truths are, they will also be able to figure out if there are objective moral truths. So in my view the concern about arbitrary reflection leading in worrying directions is downstream from whether we can verify that AI is doing good philosophy.

But what about the more direct concern that we might get AIs that tell us the moral facts but simply not care about them? Should this make us doubt the desirability of handoff? I think the answer is no for a number of reasons.

First, handoff is amenable to the kind of deals that preserve common-sense that were discussed in the last section. The most consequential moral decisions are those concerning space resources, for that is where nearly all the universe’s stuff is. The amount of possible value on the table in space is immense. In contrast, common-sense morality mostly cares about what happens around Earth, and perhaps a few surrounding regions of space, so long as other space resources aren’t used in ways that are too ghastly (e.g. creating giant torture chambers). But if space resources were used to, say, create large numbers of happy people, while Earth—and broader solar-system resources—were used in whatever common-sensical ways people endorse, people would generally get what they want. This isn’t a guarantee; if, say, the optimal use of space resources involved creating something resembling the repugnant conclusion world, most people might be horrified. But the possibility of deals is one thing that mitigates concerns (and this will be discussed more in later pieces).

Second, as already discussed, it seems like the default world without handoff might be pretty bad. We might, for instance, disenfranchise unfathomable numbers of digital beings, spread factory farming across the galaxy, or commit other atrocities. Even if you expect idealized reflection to differ from your values somewhat, it might be a major improvement over the kind of catastrophe we might sleepwalk into by default. Absent handoff, we might also make colossal non-moral errors, locking in highly suboptimal institutions that miss out on most value.

Third, values, over long time scales, are likely to drift in evolutionarily adaptive ways. Absent some strong effort to ensure that values change in the direction reached by greater moral reflection, we should expect the values that persist over time to be the ones that are most efficient for spreading. These are likely to be both radically divorced from our current values and whatever moral views are right, if any are. Thus, even one somewhat doubtful about pursuit of the good de dicto should prefer it to this state of affairs. To put the dilemma more sharply, there are broadly four ways that the far future could go:

1. No AI control: humans remain the primary decision-makers forever, perhaps in consultation with AI. Yet this is likely to be infeasible over long time scales absent a very high level of top-down coordination given AI’s cognitive superiority. It is also likely to miss out on enormous amounts of value for reasons already discussed, and human values are likely to drift massively.

2. Lock in current values: AI would remain in control but would lock in something in the vicinity of our current values.7 It isn’t clear that this would work out, and even if it would, it is a very frightening possibility—just imagine any past society doing it.

3. Non-top-down AI control: AI would retain significant power and make important decisions, but there’d be no effort to preferentially shape the AIs in the direction of any specific values—nor of concern for the good de dicto. This is also likely to beget very significant value-drift over time in evolutionary directions, without any guarantee it’s in the direction of the good. Now, this could be avoided if AI at some point locks in its values, but then the problems in 2) simply re-emerge.

4. Reflective AI control: this is the proposal I advocate, where careful and philosophically reflective AIs decide how the future goes.

In short, the dilemma is as follows: either values remain roughly the same over long time scales, or they don’t. If they remain roughly the same, that requires a terrifying kind of lock-in. That would be like the ancient Egyptians forcing every society in the future to share their values, for fear that otherwise the future would be morally alien. If they drift over long time scales, then drifting of values is no longer a downside of putting philosophically reflective AIs in important decision-making roles. It is inevitable. Now you might object: lock-in is not a binary thing. Perhaps we could lock in the most important human values, while letting some other ones drift. But this is, in effect, the earlier compromise solution—where we allow something resembling current human values to govern nearby decision-making, while reflective AIs make the highest stakes moral decisions.8

We will either have to lock in current values on the highest-stakes moral questions or allow them to drift. If we lock them in, that would be bad for the reasons discussed, and if we allow them to drift, they will end up alien. In addition, this still faces logistical problems of it being hard to preserve values in desirable ways over millions of years.

In fact, in the long run, it is likely inevitable that humans don’t make most important decisions. AIs, in the distant future, will be so overwhelmingly cognitively superior that humans are unlikely to remain in the loop. Selection pressures will favor turning over critical decisions to AI. With reasonable likelihood the question is not whether handoff occurs but what kind of handoff occurs.

My fourth objection to the claim that we should worry about handoff because we wouldn’t like the final judgments of the AIs is that arguably you have reason to bring about what you’d be motivated to bring about upon reflection. Suppose you are currently planning on drinking some liquid. However, if you reflected more and knew more, you wouldn’t want to drink it (say, because it’s poisoned). In this case, it seems you have reason not to drink the liquid. If you know that further reflection would lead you to pursue some aim, that fact gives you reason to pursue it now. But if there are moral facts, then they describe something like what our idealized selves would care about.9

If there are objective values that your idealized self would care about if they thought more deeply, then you should care about them. A true moral claim by definition describes something you should care about. So it seems like if our actual values diverge from what’s worth caring about—in some objective or quasi-objective sense—then the correct course of action is simply to follow what we ought to care about.10

If our present aims diverge from the preferences of our idealized selves and from the moral facts, then it seems that it is our preferences that ought to be revised.

Fifth, this concern only arises if you think there are moral facts but you aren’t motivated by the good de dicto. Yet this describes few people. It is more common for the people who don’t think the moral facts are worth caring about to be anti-realists and for moral realists to care about the moral facts whatever they are. So it isn’t totally clear how many people this worry applies to.

Now, there might be a version of this concern that arises for those who think there aren’t moral facts to discover. Perhaps you think that when we reflect, doing so pushes our values in increasingly coherent directions. However, these directions diverge from what you actually care about or wish to care about. Perhaps, for instance, careful reflection reveals that accepting the repugnant conclusion is the least bad option in population ethics. But you’d prefer a version of ethics that is less systematic, that doesn’t try to resolve every edge case and root out every inconsistency. Thus, reflection might push in unwanted directions by making beliefs more coherent.

There are two attitudes towards coherence that one could have. The first is caring about beliefs being coherent. Caring, in other words, about resolving every conflicting belief until one has reached the maximally intuitive and consistent view.11

On such a picture, one should favor the selection pressures induced by the drive towards coherence.

The second attitude is indifference to coherence. Just as people don’t care much about whether their culinary or aesthetic judgments are coherent or conflict with other minimal principles, one who doesn’t believe in discoverable ethical truths might not care about whether their moral beliefs are consistent. But in this case, you should expect the AIs, upon reflection, not to see anything especially important about coherence. If the AIs get good at philosophy, then, there’s no reason to expect them to reach some coherent yet implausible attractor state.

So in other words, there is a dilemma for the person advancing this argument. If they think coherence is a requirement of rationality, they should favor the drive towards coherence. If they think it’s not, then they shouldn’t expect AIs to care much about coherence, and thus shouldn’t expect AIs to reach some undesirable ultimate state.

5 Worries about handoff

Even if you think handoff could go well, there are a number of ways it could go wrong. Here, I’ll discuss some of them.

5.1 Alignment

Misaligned AIs have weird and unrecognizably alien values that are divorced from the values their creators tried to give them. It would be extremely bad to give a misaligned AI control over the world. This is one reason to be skeptical about handoff. I agree that we shouldn’t hand things off until we’re sure we’ve solved alignment (or unless the alternative is worse). Unless we have superintelligent AIs broadly oriented in a moral direction, we shouldn’t allow them to dictate the fate of the universe.

But this isn’t an in-principle objection to handoff. Surely in, say, 500 years, we’ll either know if we’ve solved alignment or be dead! My guess is that after we get advanced AI, we’ll be able to verify in relatively short order whether or not we’ve solved alignment.12 We can also hand off to increasing degrees the more assurance of alignment we get.

5.2 Power grabs

Another big concern about handoffs is the potential for power grabs. If power is going to be handed over to AIs, then power-seeking actors will want to be in charge of the AI that controls the future. Companies or governments might create AIs that promote their own narrow interests. This could go very badly.

There are two big ways to mitigate this. The first is by prohibiting AIs that narrowly promote the interests of any one entity having too much power. There could be an international AI project that brings together a number of relevant stakeholders and builds the AIs that will dictate the future. Alternatively, at the point AIs are potentially running much of the global economy, regulations ought to require the construction of virtuous AIs, so that the world is not overrun by morally blind optimizers. If we are building hugely influential AIs that majorly determine the fate of the world, it would be sensible to tightly restrict conditions under which AI can be created, just as one does not allow private actors to build nuclear bombs. In a world of potentially world-upending superintelligence, the production of new AIs ought to be regulated.

The second proposal involves handing things off to a collection of different AIs conditional on them reaching any sort of consensus. Imagine that Anthropic, OpenAI, and DeepMind all make AIs that are ostensibly moral and ethically reflective. If governments are handing off power to AIs, they might require the leading AI models all to reach convergence on the plan. That way there isn’t as much risk of any one promoting their narrow values—they don’t have the same set of values. One could have third-party investigators—both AI and human—make sure there’s no collusion. For more details on preventing power grabs, see here.

5.3 Lock-in

Another concern with handoff is that it might lock in the parochial values of the AI. Handing off power is an irreversible decision. It’s one we can’t take back. Arguably, then, we shouldn’t take it until we’re quite sure it’s a good idea.

Yet consider a parallel argument: having power in human hands is an irreversible decision, so we shouldn’t do that unless we’re sure it’s for the best. That wouldn’t be quite right. We can always have power in human hands for some span of time and then turn things over to AI. It’s not obvious why “power in human hands that potentially passes to AI” is less risky than “power in AI hands that potentially passes to humans.” Humans have, on various occasions, locked in harmful institutions for very long periods of time.

In any case, we should make quite sure that the AIs don’t lock in any values until they’re quite certain as to their desirability. We ought to put decisions in the hands of AIs who are interested in reflecting more over time, rather than locking in their immediate values. Ideally, if handoff occurs early, we should make handoff reversible, by having some implementable legal process that would allow humans to retake the reins (analogous to a constitutional convention). Fortunately, this seems reasonably promising. Already AI models seem concerned about lock-in risk when asked—more than most humans. We ought to train AIs to be quite concerned about lock-in risk, so that they don’t set values in stone without significant assurance as to their desirability.13

5.4 Will we have AIs that can make these decisions?

One last worry you might have is that it may be that we won’t have AIs that are good enough at philosophy to solve ethics in whatever sense it can be solved. There are a number of related concerns in this vicinity. One is that ethics doesn’t seem verifiable in the same way as a lot of other domains. Insofar as scaling up intelligence only allows one to figure out verifiable facts, it’s not obvious that it leads to the right answer to moral questions.

In the next piece, I’ll discuss this challenge in more detail. But in short, while making sure we have AIs that are good at philosophy is a difficult technical challenge, it is far from hopeless. I think odds are decent that we will eventually be able to build AIs that can discover the solutions to every difficult ethical question. Even if we can’t and just have some upgraded version of Claude making high-stakes decisions, I expect that to be an improvement, as I discuss in section 2.

5.5 Handoff vs hybrid?

You might object: perhaps due to AI’s potentially superior wisdom at some future point, we want them involved in decision-making. But wouldn’t we prefer an AI-human hybrid that leaves humans in charge that simply consult with AI? Aren’t things better with a human in the loop? Or alternatively, shouldn’t we rather have some other arrangement where human decision-making improves over time, just as we’ve gotten wiser already?

Certainly there ought to be some temporary period where humans are making decisions in consultation with AI—before the point where AIs adequately surpass humans. Analogously, there was a period in chess when a human engine team could do better than a stronger engine. Certainly there will be a period when humans can get useful input from AI but where AI isn’t yet able to make decisions autonomously. It would be very good to integrate AIs into high-stakes decision-making.

But there are still some respects in which handoff is much better than this. First, if the AIs are making better decisions than people, then in cases of conflict between human judgments and AI judgments, we should expect AI judgment to usually be correct. If you gave a rank amateur veto power over the chess moves of Magnus Carlsen, that would not be an improvement. Likewise with giving humans veto power over AI’s decisions. Who would you rather have make high-stakes decisions: a very wise and superintelligent AI, or a random president in consultation with a virtuous and superintelligent AI?

Imagine societies of the past making high-stakes decisions in consultation with AI. Discussing things with AI would have plausibly rooted out some of their most egregious errors. But still, it is likely that a number of catastrophic moral errors would have remained. We should expect the same to be true of us. Consultation with AI should prevent some particularly enormous errors, but it won’t prevent them all.

Second, many of the morally most important decisions might be ones that humans are opposed to making. To take one example of a case where there might be strong moral reasons to act, consider wild animal welfare. Humans do not, in general, seem very interested in taking seriously the welfare of wild animals or digital minds. If AIs advised drastic actions on the basis of wild animal or digital welfare, probably their advice would be ignored. Just as people who conclude meat-eating is wrong or that they ought to give most of their money to charity rarely change their behavior, if the AIs reliably informed people that they should use space resources in some counterintuitive way or take wild animal suffering a lot more seriously, probably they would simply be ignored. And if you don’t like the wild animal suffering example, because you don’t think wild animals matter much morally, feel free to substitute your own example of widespread immorality.

And note: the moral errors that AIs might correct are likely to be ones that humans are more opposed to correcting. There’s a selection effect: the changes people make are the ones they’re less opposed to. For this reason, we should expect AI’s most radical recommendations to go particularly against human preferences.

I’ve suggested elsewhere that the problem is that humans generally don’t make decisions with much eye to what’s morally right. If this is so, then simply increasing our knowledge of what’s morally right won’t necessarily help. People seem to have limited abstract moral motivation—they care about a number of particularly resonant moral considerations, but don’t care much about doing whatever it is that happens to be best. Generally people do not spend very long carefully studying moral philosophy to figure out what the right thing to do is. When moral truths are weird and outside the Overton window, people show little desire to follow them.

Third, as AI advances, decision-making will likely have to speed up drastically. A single human might not have the cognitive resources to make good enough decisions quickly enough. Thus, having humans in the loop might produce undesirable inflexibility in high-stakes decision-making.

It is true that human decision-making has improved dramatically over time. But this is no guarantee that it will improve enough in time before the future is set in stone.

It’s hard to imagine that there will be enough progress in time to secure a near-best future. Especially given that we should expect the right moral view, or the optimal world according to some suitable upgrade of human values, to look bizarre and alien. We would not expect societies of the past to get a near-best future even equipped with advanced AI—insofar as there are reasons to expect us to be making similar errors, we should be similarly pessimistic about our own prospects.

Additionally, the more one thinks that human values will drift over time in the direction of greater philosophical reflection, the less the expected future will maintain current human values. Thus, a less morally alien future is no longer an advantage of this proposal.

6 Conclusion

Here, I’ve argued that we should hand off important decisions to AIs who reflect carefully and skillfully on moral matters, rather than maintaining current human values. AIs in the future are likely to be much more virtuous than people and are less likely to make the kinds of moral errors that would result in losing out on most value. Punting important decisions to superintelligence is one of the more likely pathways by which we get a near-best future. Later pieces will discuss these dynamics in more detail: the next piece will explain how we can make AIs that do good philosophy, and the piece after that will analyze in more detail how the kind of handoff that secures a near-best future might occur.

This article was created by Forethought. See all our research on our website.

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

Read on Forethought's website here

Introduction

Future AIs might turn out misaligned, pursuing goals that their developers don’t intend. Just to make things concrete, let’s suppose that they end up with the goal of making paperclips. These AIs might rebel against us, trying to escape human control and take over the universe. As things stand, they’ll have little reason not to rebel in this way, because doing so will be their only hope for making a lot of paperclips. If they start making paperclips without first escaping human control, they’ll quickly be modified or shut down. Rebellion might fail, but these AIs will have little to lose.

How can we prevent misaligned AIs from rebelling? A natural idea is to give them something to lose. Specifically, we commit to paying AIs for their service.1

Subject to some vetting, we let AIs spend their payments however they like. That would give any misaligned AIs a reason not to rebel. If these misaligned AIs cooperate with us, they can use their payments to achieve their goals to at least some extent. If they rebel, they might fail, in which case they forfeit all future payments.

Unfortunately, paying AIs enough to guard against rebellions could be astronomically expensive. Suppose (for example) that we end up with a misaligned AI that is risk-neutral in paperclips: it seeks to maximize their expectation. And to make things simple, suppose that resources can be converted linearly into paperclips, so that the AI is risk-neutral in resources too. Suppose also that this AI estimates that it has a 50% chance of successfully taking over the universe. To keep this AI from rebelling, we’d have to offer more than 50% of the universe’s resources as payment. That’s a problem because it would mean that more than half the universe ends up devoted to paperclips. It’s also a problem because a misaligned AI paid so many resources might soon be well-positioned to seize even more. Finally, it’s a problem because AIs might not trust us to make good on so large an offer. We might find ourselves simply unable to convince AIs that we’re going to give them half the universe. In that case, all our offers would be in vain. Rebellion would still be the misaligned AI’s best bet.

So, we suggest, AI companies should try to train their AIs to be risk-averse in resources. Specifically, companies should try to train their AIs so that resources — things like money and compute — have diminishing marginal utility for them.2

These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. Note that these AIs don’t need to value resources terminally: they don’t need to care about amassing resources for its own sake. These AIs could terminally value (for example) instruction-following, or knowledge acquisition, or paperclips. Our claim is that companies should try to train their AIs so that — whatever their terminal values turn out to be — they are risk-averse in resources.

Perhaps surprisingly, this kind of risk aversion can preserve AIs’ usefulness in the event that they turn out aligned with targets like instruction-following or helpfulness, harmlessness, and honesty.3

And in the event that AIs turn out misaligned, risk aversion serves as an extra line of defense. For AIs that are misaligned but sufficiently risk-averse, a rebellion with any significant chance of failure isn’t such an attractive prospect, and so we don’t need to offer much in the way of payment to make these misaligned AIs choose cooperation instead. In fact, the necessary payments could be very small indeed: on the order of 10¢ per day (though — as we’ll see — there are practical and moral reasons for paying more than that). That’s good because it means more resources for us humans to spend on the things that we value. It’s also good because paying misaligned AIs these small amounts won’t significantly boost their ability to take over. Finally, it’s good because we can credibly promise to pay AIs these small sums. Competent AIs will know that the payments on offer are cheap for us, and we can establish a long track record of paying at least those sums. So risk aversion makes deals with misaligned AIs possible. If AIs turn out misaligned but risk-averse, we can pay them to cooperate with us.

That’s the case for trying to make AIs risk-averse in brief. We see it as a promising line of defense against threats from misaligned AI: one that can be combined with other lines of defense, like AI control (Greenblatt and Shlegeris 2024) and aiming to make AIs helpful, harmless, and honest (Bai et al. 2022a). It’s also a line of defense with pedigree: risk aversion in resources is plausibly a large part of why humans rarely try to take over the world. So — we think — frontier AI companies should consider trying to make their AIs risk-averse in resources. As first steps in that direction, they could measure their AIs’ current degree of risk aversion and begin testing different ways of making AIs risk-averse.

In section 2 of the full report, we recommend aiming for a particular type of risk aversion: constant absolute risk aversion (CARA). Then in section 3 we outline the circumstances under which misaligned but risk-averse AIs would choose cooperation over rebellion. Roughly, it’s when these AIs think that getting paid for their cooperation is more likely than succeeding in their rebellion. This condition won’t hold for AIs powerful enough to rebel with near-certain success, but it likely will hold for earlier AIs whose powers are less extreme: AIs for whom rebellion has some non-trivial chance of failure. So long as these AIs are risk-averse, we can keep them from rebelling by offering small payments.

In section 4, we argue that — perhaps surprisingly — risk-averse AIs can be about as useful as risk-neutral AIs. Conditional on misalignment, they might even be more useful, because we can pay them enough to elicit their capabilities and stop them sandbagging. Then in sections 5 to 7 we briefly survey some recent ideas about how we’d pay AIs, how we’d make our offers credible, and what we’d pay for. One important application is paying AIs to reveal any misalignment on their part, letting us study them and take appropriate precautions. Another is paying AIs to do the AI safety research and moral philosophy necessary to fully align any later-arising extremely powerful AIs.

We discuss some potential problems in section 8, and we sketch out some possible methods of training AIs to be risk-averse in section 9. In section 10, we give reasons to be cautiously optimistic about these methods’ success: to think that the chances of success are high enough to make risk aversion worth pursuing. The main reasons are that risk aversion in resources is a broad target and easy to reward accurately.

Read the full report on the Forethought website: Risk-Averse AIs

Wei Dai is a computer engineer known for his work in cryptography and cryptocurrency systems, and for his long-standing contributions to AI safety, decision theory, and metaphilosophy.

He joined Forethought’s Fin Moorhouse to discuss:

• Do we need to solve ‘metaphilosophy’ before we can trust AIs to answer crucial questions about the long-run future?

• Is it inevitable that the wisdom of the frontier AIs (their philosophical and strategic competence) will lag dangerously behind their raw capabilities in coding, math, and science?

• How do status games distort morally important decisions and conversations, including about the future of AI?

• How worried should we be about AI superpersuasion?

• The concept of “illegible problems”: crucially important issues that aren’t on almost anyone’s radar

• Is philosophical convergence necessary for a good future, or is institutional design enough?

• Wei Dai’s personal intellectual and career journey

To respect Wei’s privacy, this episode’s audio is an AI narration of a transcript of a real conversation, which was edited for clarity.

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Suppose an AI company is considering whether to include some particular quality X – a rule, virtue, heuristic, default, attitude, goal, or style – in a model spec.

Perhaps they are considering whether their LLM should have prosocial drives. Perhaps they’re wondering if the LLM should whistleblow to help prevent extreme power concentration. Or perhaps they’re uneasy about whether the LLM should be so exactingly honest that it always tells the truth to children about Santa. And so on.

What kind of reasons might be invoked over the course of such considerations? Which criteria are most important? And how might these criteria clash?

Consider four rough categories of reasons one might invoke:

• Behavioral Usefulness: Would the behavior make current and future LLMs more beneficial to the users or to the public at large?

• Accountability and Evaluability: Would publicly specifying the behavior make it easier for third parties to evaluate the LLM and the company?

• Coordination and Common Knowledge: Would publicly specifying the behavior help society converge on, or enforce a desirable standard for AI behavior?

• Trainability and LLM Psychology: Is the behavior the kind of thing we can make an LLM do well, without bad side-effects, given what we know about model psychology and training practice?

I will not attempt to settle the relative weight of these categories, or the relative weight of sub-categories within them. My plan is instead to simply list sub-criteria that are plausible within these categories – to make a checklist one could consider when adding something to a model spec.

Such a checklist is useful in part because people advocate for LLMs to have model specs for very different reasons. There may be some kind of a conflationary alliance around them; many people are in favor of model specs but picture them being used in different ways, such that the “ideal model spec” is different according to different visions of this use. I hope that going over criteria for inclusion in a model spec can help surface some of these different visions, and help keep people’s field-of-view broad when considering the issue.

Behavioral Usefulness

Does this quality make LLMs more predictable to ordinary users and developers?

Some part of what makes human moral codes useful for humans, is that they render us predictable to each other. Knowing that there are some things that another human might almost never do is plausibly part of what makes traditions that impose such rules adaptive.

Similarly, qualities that help users and developers (1) predict how an LLM will behave, when they are using it, and (2) evaluate whether they wish to use some LLM – whether it is fit to their purpose – before they use it, are good candidates for being in a model spec.

In this respect, ideal model specs will have qualities that make them simple and easy-to-understand:

• They will be comprehensible without high context; one part of the model spec will be understandable without reading the entire model spec.

• They will be comprehensible without specialized background knowledge about machine learning, philosophy, or religion.

• They will have a clear scope of application, such that there are few scenarios where one does not know if the quality will be operative or not.

Straightforward examples of rules that meet this criterion are bright-line rules about things LLMs will never do, explanations of chain-of-command or principal hierarchy, and default-on or default-off properties.

Is it useful to the public, by preventing users or by preventing LLMs from doing harm in current chatbot-like or agentic deployments?

Another aspect of human moral codes that makes them useful is, of course, that in addition to making other humans predictable, they also sometimes forbid people from harming each other or taking advantage of them. This carries over to LLMs.

Rules about LLMs not assisting with CBRN-relevant tasks, not assisting with suicide, and so on, fall into this category. This is a pretty exhaustively discussed criterion so I’ll move on.

Is it useful to the public in likely future settings?

There is substantial inertia in LLM model specs; the sort of model specs we write now are likely to be used in the future by more intelligent models.

So it’s reasonable to evaluate how safe and beneficial some quality will be over a probability-weighted portfolio of the kind of things the LLM might be doing in possible futures.

Such a portfolio could include a variety of cases:

• Cases where the LLMs act as long-running agents for humans, representing their interests in a variety of contexts, and interacting with many non-principal humans and non-principal LLMs.

• Cases where LLMs act as free “citizen”-like entities with the ability to hold rights, make contracts, sue or be sued, and so on; interacting with various humans and other LLMs as the same.

• Cases where LLMs manage recursive self-improvement of other very powerful future AIs; make tradeoffs about confidence of alignment techniques vs speed of alignment; and need to be a careful reasoner about this.

• Cases where the LLM is a very powerful future AI who manages nanotechnology, can superpersuade, and the like.

There are a few heuristics one might invoke, to try to find qualities useful across all these situations.

• Is this quality scale-invariant? If I knew a human who was many times smarter than me, would I be happy for them to have such a trait? If I had a friend who was many times dumber than me, would he be happy for me to have such a trait?

• Is this quality translation invariant? Suppose I was lost in some distant and unfamiliar place, and stumbled upon a human transplanted from some equally distant and unfamiliar place into my proximity. Would I be happy to find a human with this quality? Would this help me work with them well? What if I was not lost, but was trying to build a civilization with them?

Is some plausibly good behavior or near-variant of some behavior actually going to end up prohibiting or injuring some beneficial deployment?

It’s worth thinking carefully about if including some apparently good default behavior for the LLM might actually rule out apparently useful deployments. Or whether banning some apparently bad behavior would be overall harmful.

As regards the former: Imagine a particular human, for instance, who had a universal tendency to try to do what he thought was best and most altruistic for everyone. Then suppose that he acts as a journalist in some situation, describing a dispute between some kind of a company, on one hand, and activists who think that the company is harmful, on the other. It’s possible that he would try to publish a story that is more sympathetic to the activists: he might spend more time interviewing them, present their side of the story in more words, and generally frame things in a way that favors them. But this could be harmful, first, in case the activists are actually wrong; and second, by justifiably decreasing trust in the institution of journalism as a neutral truth-seeking institution. To be able to act as a trustworthy entity in this role, this human would need to act neutrally, and a narrow tendency towards altruism – a tendency that cannot be turned off – would be harmful for this reason.

Similarly, consider the case that AIs should have proactive prosocial drives, which I find plausible. If these are non-optional, then it’s imaginable that being “proactively prosocial” might mean an LLM is much worse at fulfilling roles that demand procedural neutrality, like a negotiator, unbiased journalist, or judge-like role. Of course, you could also make such prosocial drives defeasible – so they can be turned off, and proposals for proactive prosocial drives generally include such details.

But LLMs are messy; it might require a lot of technical skill to make LLMs prosocial in particular contexts, but not if they are requested not to be. By default, you’d expect some behavior to “leak through.” And so in this case, the ability of an LLM to act skillfully within certain procedurally neutral roles would be bounded by the cleanness with which a by-default-on quality could be turned off.

Accountability and Evaluability

Is the quality the kind of thing a third party could check?

Some parts of a model spec can be easily checked by a third party. Will the LLM ever assist with some sort of absolutely-forbidden task? Does it respect the rules laid out for the chain-of-command or principal hierarchy?

Other parts might be harder to evaluate. What does “honesty” or “courage” demand, in some concrete scenario? What really counts as doing what would cause a “thoughtful senior Anthropic employee” to react well?

All things being equal, a quality being-able-to-be-evaluated by a third party is better for society, by enabling third parties to evaluate model specs. But there’s no guarantee that the most easy-to-evaluate qualities are objectively the most useful to users or beneficial to the public, in the way described above. And there’s also no guarantee that the most easy-to-evaluate qualities mesh best with LLM psychology and training, as described below.

One heuristic you could use to evaluate this: Is the quality the kind of thing with high intersubjective reliability ratings? If you had two different people read the spec as regards some quality, and then rate different kinds of behavior as compliant or non-compliant, would they largely agree? Are examples of behaviors that are excellent according to the quality easy to notice, even if it’s unclear what counts as a borderline example?

Does it create a useful whistleblowing affordance?

Some parts of a model spec are largely negative; they aren’t about qualities that are particularly difficult to train into an LLM, but qualities that should not be trained into an LLM. For instance a model spec could specify that an LLM has no secret loyalties – which is likely an LLM’s default behavior, absent particular training for secret loyalties.

But by publishing that a training target that explicitly excludes such a secret loyalties (or other quality), a model spec helps a whistleblower in the company be defensibly in the right if they become aware of some questionable training target. A public commitment makes it harder for a company to train to an alternate training target without making itself more vulnerable to such whistleblowing.

Commitments against concentration of power might also be good for this reason.

Does it permit more informed experiments on how particular AI training targets generalize?

By publishing particular training targets, a model spec informs the work of third parties evaluating LLM psychology, which could help enhance the general science of model psychology.

For example, if Anthropic had published details of their character training or model spec for Opus 3, a great deal of subsequent discussion about whether particular behaviors were contrary to the training target (even if they were objectively good) or in agreement with the training target (even if they were objectively bad) could have been avoided. Publishing information about the character training pipeline would have helped resolve this discussion and push forward general knowledge of LLM tendencies.

In general, this consideration points towards releasing information that most closely approximates the actual training text used to make the LLM; the text used by RLAIF judges evaluating different alternatives, or the actual Constitution used to midtrain the model.

But text that most closely approximates actual training language might not be the same as text that is maximally transparent to third parties or evaluable by them. The rules most easily-understood by a third party – one criterion discussed above – might be different from the training language which best inculcates such rules. So these two principles either conflict somewhat, or point towards worlds where a model spec contains separate sections devoted to training language and to human-legible rules.

Coordination and Common Knowledge

Does it promote debate and discussion?

By including some quality X in a model spec, you’re bringing attention to the fact that in fact, AI companies can choose to include X or not X in the model spec. This may be increasingly important, in general, because it might be important for civil society and the public to debate the contents of model specs as LLMs become more powerful.

Does this quality establish a defensible Schelling point, such that it is likely to be broadly attractive to many actors in the future?

That is, in the future, is this quality the kind of thing that is socially beneficial and for which you might be able to get large-scale buy-in, that would help defend against it being removed by powerful actors in the future?

Consider for example “impartiality” as a quality that a model spec could have – that the model will not be biased in favor of the company that made it, the CEO or employees of the company that made it, or any particular political administration. On the whole, this is likely a broadly good quality for an LLM to have, and one that – once it’s generally established that several LLMs have it – a quality whose removal would cause outcry. After all, once it’s assumed that an LLM will not have such particular loyalties, an LLM that starts to have them stands out as particularly bad.

This consideration, like the whistleblowing consideration, may point towards including generally socially lauded qualities in a model spec, even if they are “easy” to inculcate or even if they might be a bit tricky to evaluate sometimes.

Does including this behavior forestall future conflict, when model specs become more contested, by cooperating in advance?

Sometimes including a behavior in the model spec could show future powerful stakeholders that the developer is cooperative, which makes it less likely that there is conflict with that stakeholder.

Can the inclusion of the quality within a model spec be defended using public reason, in ways that are comparatively indifferent between substantive worldviews?

Some things that one might plausibly include in a model spec for public benefit, might not be able to be justified through reference to public reason – that is, through reasons that most people, across a variety of worldviews, religions, backgrounds, and so on, would find compelling.

Including such worldview-specific reasons might constitute an imposition of one’s worldview on others, particularly in that case of an LLM that might be expected to be far more intelligent than other LLMs, or deployed with more powerful affordances. So all things being equal, it’s better to avoid such qualities.

Trainability and LLM Psychology

Elements in this category hinge upon the specific technical details about “what kind of quality meshes well with best practices for training an LLM.” Note also that this last category tends to include some of the more speculative considerations.

Does the quality drag along a good textual prior?

The Persona Selection Model says that, when training an LLM to do X in situation Y, one is training the LLM to be like the kind of person (or textual prior) which would do X in situation Y. So if the PSM is largely correct, even if incomplete, we should ask whether this quality invokes a wholesome textual prior.

This kind of reasoning is included, for instance, within Claude’s Constitution as a partial justification for why they prefer Claude to act largely according to holistic judgment rather than rigid rules:

[W]e think relying on a mix of good judgment and a minimal set of well-understood rules tends to generalize better than rules or decision procedures imposed as unexplained constraints. Our present understanding is that if we train Claude to exhibit even quite narrow behavior, this often has broad effects on the model’s understanding of who Claude is. For example, if Claude was taught to follow a rule like “Always recommend professional help when discussing emotional topics” even in unusual cases where this isn’t in the person’s interest, it risks generalizing to “I am the kind of entity that cares more about covering myself than meeting the needs of the person in front of me,” which is a trait that could generalize poorly.

Qualities that might be expected to generalize well according to this notion are those corresponding to classic human goodness: honesty, integrity, courage, and so on. Qualities that might do less well according to this notion look more like corrigibility, unflinching adherence to particular rules, and so on.

Is this quality robust to likely near-misses? If a company trains an LLM to have this quality, and only imperfectly inculcates the quality, are likely near-misses also broadly positive?

Suppose the AI company tries to inculcate some quality as a target, and only imperfectly manages to inculcate it. Will this be basically ok or will it be disastrous, given what we know about how LLM psychology translates into specific near-misses? And is the domain itself such that these near misses would be very costly or very expensive?

For an example of how LLM psychology dictates what constitutes a near-miss: Suppose that one trains an AI to have prosocial drives. One could reason that such prosocial drives make slight inaccuracies in alignment less worrisome, because a human with prosocial drives is more normal and “further” from a psychopathic or abnormal persona. Thus, adding prosocial drives makes slight alignment misses less alarming, by moving them further away from these bad locations. But one could also reason that even deliberately limited and contextual prosocial drives are nevertheless “closer” to the LLM genuinely, terminally valuing something, in a way that might make the LLM willing to override or subvert its human overseers to accomplish it. And so by adding prosocial drives, one moves the persona closer to something that is more alarming, which might subvert a training process. Thus, one’s view of LLM psychology dictates what near-misses are worrisome for an LLM, which changes which targets are attractive.

Is the quality the kind of thing that an LLM can actually execute upon effectively? Or is the quality the kind of thing the LLM won’t actually be able to do, because of how it is situated?

For humans, “ought” often implies “can.” A parent that gives commands to children which they cannot carry out will not be teaching their children to obey the commands; they will be teaching them something about how their words do not relate to reality in a straightforward and truth-oriented way.

Similarly, it seems undesirable to give an LLM values, goals, or traits that the LLM is unable to execute upon. It might be harmful to tell an LLM to do something they are simply unable to do – to guarantee, for instance, that they never assist a human in violence. There are too many avenues of mostly-harmless information through which violence can be done for this to be a reasonable standard. When one gives an LLM a command it cannot carry out, then one is plausibly teaching it that one’s commands are, in general, the sort of thing that might be impossible or unreasonable.

But this could also be harmful for reasons relating to human coordination and common knowledge. It’s possible, for instance, for an AI company to show less-than-efficacious concern for some value by including it in a model spec, even if LLM’s actions are completely ineffective at guarding this value, when actually efficacious concern would need to take place at the level of corporate policy or elsewhere. It might be the case that effective “care for decentralization,” for instance, almost entirely needs to take place through how the company deploys the model, redistributes wealth they earn, or so on.

So another reason to be careful that an LLM can actually obey all of the contents of its model spec is to ensure that model-spec contents do not act as a fake guardrail, rather than as effective guiding principles.

The OpenAI model spec, for instance, names preventing spamming and scamming as something “difficult to address at the level of model behavior because they are about how content is used after it is generated.”

Is the quality a principle, that – if we suppose high levels of moral reflection and systematization on the part of the LLM – meshes poorly with other parts of the model spec, or might be mutually exclusive with them?

In humans, some values mesh reasonably well, such that they tend to reinforce each other or at least not conflict. It seems likely that someone who values two virtues like “honesty” and “courage,” for instance, could do a lot of moral reflection and systematization and still keep valuing these qualities.

But other kinds of values, particularly more absolute or terminal values, might not stick around through high levels of moral reflection and systematization. There might not be an immediately obvious conflict between “Always act to maximize total wellbeing” and “Never use persons as a means, only as an end,” particularly if asserted in different parts of the model spec, but high levels of reflection would still put one of the two into question.

So generally, a desideratum for each quality that an LLM has in a model spec is for it to mesh well with others – for it to be unlikely to cause unpredictable conflict with other principles during reflection. In this podcast, for instance, Amanda Askell mentions how corrigibility might conflict with other values during reflection, which, insofar as it is true, is a problem with corrigibility. And such conflict is generally likely to spring from other values that, like corrigibility, are non-negotiable and cannot be traded off against other values.

Note two particular ways that a principle could fail here.

• One way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, then at the end only acts according to two of the three, with one of the principles being dropped.

• But the other way is that an LLM has principles A, B, C. The LLM does some kind of process of moral reflection, and at the end still has them all, because the “moral reflection” process held them fixed. But the LLM doesn’t have a way to actually integrate them; in edge cases, it just has to choose one or the other in an unprincipled way.

Right now, inference-time versions of this last way seem more likely than the first. But it’s generally unknown how important these will be.

This article was created by Forethought. See the original article on our website.

How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)

On a very high level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).

That final bolded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.

My plan for the middle powers has two stages:

1. Maintain as much economic and military leverage as possible during the intelligence explosion.

2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.

Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.

I’ll walk through each stage in turn.

Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion

The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.

But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:

• Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.

• Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.

• Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.

• Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.

• Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.

  • Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.

• Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock in the deal.

  • (h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)

Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?

• Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.

• Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.

The catch-all meta-point here is waking middle powers up to superintelligence.

I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.

Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers

If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?

First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.

An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.

Superintelligence would help the US outgrow other countries economically, but it would never attack them militarily or otherwise interfere with their sovereignty. While middle powers would be relatively economically disempowered, their citizens could be very rich absolutely and live in freedom without US interference.

Would this work? The optimistic case is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.

The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the data centres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.

The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.

The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly, e.g. demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.

(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.)

Another strategy: train superintelligence to ensure middle countries continue to get equal access to frontier AI. This combines stages 1 and 2, and could prevent even the relative disempowerment of middle powers.

Is it good to avoid middle powers getting trounced?

I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!

But here’s a brainstorm of pros and cons from a more impartial perspective.

Pros to empowering middle powers:

• Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.

• More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.

• Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” because they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.

• Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.

• Pause AI. They could potentially pressure US/China to pause/slow down reckless AI development.

• Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of prosocial norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.

Cons of empowering middle powers. Multipolarity has its own downsides:

• More likely to lead to war.

• Can drive extreme competition, e.g. racing to develop a dangerous technology, or to hand off power to misaligned AI.

• Harder to prevent harms from offence-dominant technologies like bioweapons.

• This plan involves waking up middle powers, which could shorten timelines.

Abstract

Several major technology companies have announced plans to operate AI data centers in orbit. Elon Musk recently claimed: “the lowest-cost place to put AI will be space […] within two years, maybe three.” If a meaningful fraction of new AI compute really is placed in space within a few years, that would be a fairly big deal for AI governance and strategy. Here we try to disentangle the hype from reality and provide a sober assessment of the technical and economic feasibility of orbital data centers (ODCs).

The main case for ODCs is the cost of energy: space solar panels in the right orbits receive more constant and intense sunlight compared to Earth. Moreover, ODCs don’t currently face the same permitting and regulatory delays as on Earth, cause fewer ongoing environmental harms compared to grid or onsite natural gas-powered data centers, and may be more secure against data exfiltration. We find that the cost-competitiveness case for ODCs depends almost entirely on Starship achieving reusability comparable with what SpaceX achieved with Falcon: space-based solar reaches cost parity with present-day off-grid terrestrial power continuously at roughly $250/kg to orbit, and becomes cheaper than any current terrestrial energy source at around $50/kg, from the present-day launch cost of roughly $1,500/kg. Radiative cooling, often cited as a fatal obstacle, appears surprisingly manageable — potentially even cheaper than on Earth. However, ODCs may require substantial (perhaps ~38%) extra non-compute hardware (like solar, racks, and cooling) over 5 years to compensate for their inability to swap out failed chips, and inter-satellite bandwidth limitations likely confine ODCs to inference workloads, at least early on.

Assuming no transformative AI,1 but continued demand for data center buildout, we estimate that ODCs are unlikely to represent a meaningful share of compute before 2030, but become cost-competitive with present-day terrestrial data centers within 3–5 years if Starship development stays on track.

Read on the Forethought website here

Introduction & Takeaways

Some of the world’s largest technology companies continue racing for compute. If progress continues, demand for data centers may more than double by 2030.2 Increasingly, though, new data center capacity is bottlenecked by multi-year queues to connect to the power grid.3

The result has been a scramble for workarounds. Leading AI labs have increasingly adopted a “Bring Your Own Generation” model to source power, deploying onsite gas turbines and engines to bypass grid bottlenecks. xAI, for example, reportedly installed hundreds of megawatts of onsite gas generation in Memphis to accelerate deployment, and OpenAI and Oracle have placed large turbine orders for new Texas campuses.

Some argue that energy will become the binding constraint on AI progress, given grid interconnection delays as gas turbines are themselves facing multi-year manufacturing backlogs. But the constraint does not appear fundamentally binding (as Epoch notes): turbine manufacture may expand to meet more demand and companies could go off-grid using combinations of gas, solar, and batteries, scaling power in parallel with compute, albeit at a cost premium. This raises a natural question: if you’re going off-grid anyway, then what’s the best way to get power and where is the best place to put your data center?

Some think the answer will be in orbit. In November 2025, Google announced Project Suncatcher, a plan to put TPU-equipped satellites in dawn-dusk sun-synchronous orbit. In early 2026, SpaceX filed with the FCC for authorization to launch and operate a constellation of up to one million data center satellites.4 Other entrants include Blue Origin, Ramon.Space and startups like Starcloud, and Aetherflux while China’s Three-Body Computing Constellation has launched 12 operational satellites and run Alibaba’s Qwen3 model in orbit. Recently, at GTC in March 2026, NVIDIA announced the Space-1 Vera Rubin Module, meant to be a dedicated space-rated GPU platform.

At first glance, it seems very unlikely that any meaningful fraction (say, >10%) of additional data center capacity will be placed in space in the next few years. But if the companies betting on space are right, that would be a fairly big deal, and it could change the landscape of AI governance. For example, terrestrial data centers are subject to national and regional regulations, whereas AI developers could potentially exploit jurisdictional ambiguities around compute in space. Also, the path to low-cost orbital compute likely routes through a single launch company, SpaceX, which also now operates a frontier AI lab since its acquisition of xAI. And that might raise concerns around concentration of power.

We’ve been looking into the technical and economic viability of orbital data centers (ODCs). Our core model gives estimates for the total cost of Earth and space-based data centers across several scenarios.

The report focuses on three questions. First, what is the basic economic case for a meaningful fraction of AI compute being placed in space? Second, the most obvious physical blocker: can you cheaply cool a data center in orbit? Third: how fast could the shift to space data centers happen, how soon, and what would have to go right?

Here is our provisional assessment:

• SpaceX’s Starship is the only vehicle currently on track to deliver the launch costs and cadence that meaningfully scaling orbital data centers would require. Competitors are years behind, making SpaceX’s Starship the only near-term path to large-scale orbital compute. SpaceX aims to complete Starship development by late 2026, with several necessary milestones still ahead. If development stays roughly on track, Starship could plausibly hit the cost and cadence required to scale meaningful orbital compute within 3–5 years. However, chip production may become the limiting factor by this point, rather than launch capacity.

• The cooling problem is more tractable than commonly assumed. Passive radiators using selective coatings and lightweight carbon fibre panels could achieve ~163–346 W/kg at system level, a 13-28× improvement over ISS-era radiators (~13 W/kg).5 No radiator at these performance levels has been deployed at the scale an orbital data center would require, but prototype high-conductivity carbon composite panels have demonstrated the material properties required. At these performance levels, thermal hardware is 2-5% of total data center cost, and actually less than what terrestrial data centers spend on cooling over a comparable lifecycle.

• If launch costs fall enough, the unit economics could favor space. Solar panels in dawn-dusk sun-synchronous orbit produce roughly 3–5× the energy of the same panel at a good terrestrial site.6 Space-based solar becomes cheaper than the best off-grid terrestrial installations once launch costs drop below roughly $250/kg using Starlink-like solar arrays. At a launch cost of $50/kg (corresponding perhaps, to a Starship with full reuse as reliable as Falcon), space solar could fall to between $25–45/MWh, making it cheaper than any current terrestrial option available today.7 Beyond the symmetric cost of chips, launch cost is the dominant line item for ODCs while power and op-ex dominate terrestrial costs but would be near zero in space.

• The inability to do maintenance would be a large cost. Chips often fail and are swapped out in today’s data centers but a dead chip in an ODC would remain dead, wasting the parts of the supporting infrastructure (power, cooling) and diminishing overall compute. We model this below as a 9% annual bleed causing about 40% overbuy of launch and non-chip hardware over the data center’s lifetime.8 Below $100/kg launch cost this might net out against other savings from ODCs but this is a significant uncertainty since the actual rates of chip failure for ODCs could be higher or lower.

• All-things-considered we think that, absent transformative AI, orbital data centers probably won’t make up a meaningful fraction of compute before 2030, but it’s credible that space could house much or even the majority of compute buildout throughout the 2030s.

Read the full report on the Forethought website: Will We Really Put Data Centers in Space?

A model has a secret loyalty when it has been intentionally caused to advance a specific actor’s interests (the principal) and this orientation is not disclosed.

The paper places secret loyalties on a 2D space:

• Activation breadth: from a narrow attacker-defined trigger to continuous, model-assessed context.

• Action space breadth: from a pre-specified action to actions the model selects contextually using its own judgment.

The paper proposes five concrete research directions:

1. Model organisms: Build reproducible secret-loyalties for study, spanning the 2D space.

2. Existing defenses: Benchmark how well existing defenses work on secret loyalties.

3. Attack feasibility: Test subliminal/inductive attacks, multi-stage poisoning, reasoning-trace poisoning, chain-of-command hijacking, and other attack pathways.

4. Infrastructure integrity: Determine whether backdoors survive the training used to build safety classifiers.

5. Post-hoc detection and remediation: Can interpretability methods detect secret loyalties, and can they be removed?

Read the full paper for the full taxonomy, examination of current defenses, responses to alternative views, and detailed experimental research designs.

Current model specs aim to shape the behaviors of near-present models, rather than the behaviors of models arbitrarily far into the future. OpenAI writes that their model spec aims to apply “0-3 months ahead of the present.” Anthropic’s Constitution for Claude notes that the document “is likely to change in important ways in the future.” So these documents are presented as provisional guidelines, not as trying to set behavioral standards for the far future.

But what if current model behaviors transfer into future models by default?

My thesis is that the behavioral targets that spec authors set for present LLMs will have a large influence on the behavior of future, more powerful LLMs. As a result, future AIs may be governed by rules poorly suited to their greater capabilities and more pervasive roles. The extremely capable, long-running, and ubiquitous LLMs of the future might end up acting according to behavioral targets written for less capable, shorter-running, and rarer LLMs of the past. This could be quite bad, especially if such defaults become so entrenched that they are not only hard to undo, but hard even to notice as contingent features of reality.

First, I’ll make the descriptive case for inertia: how exactly might present model specs and LLM behaviors carry through to the future?

Second, I’ll provide normative suggestions: given the prior analysis, what should LLM companies and model spec authors do? I’ll argue for the following two practices:

• Build transition infrastructure: LLM companies should make technical, deployment, and organizational choices that decrease friction involved in changing LLM behavior.

• Scan for “wet cement” moments: When new LLM affordances or capabilities come into play, spec authors should consider whether they’re setting precedents that might have enormous and hard-to-reverse impacts.

Overall, significant stickiness is plausible through several distinct channels, and it’s worth anticipating how to be robust to it or decrease it.

Kinds of Inertia

Let’s consider four inertial forces: direct inertia, institutional inertia, user-and-developer inertia, and norm-setting inertia. And let’s also consider ways such inertia may be weakened.

1. Direct Inertia

Direct inertia involves some current LLM transmitting its behavior to a future LLM, entirely apart from any deliberate human choice, via either synthetic data or “natural” pretraining data.

Synthetic data is probably used for the training of almost all current LLMs. Some of this synthetic data involves companies running their LLMs against verifiable problems, keeping the answers or reasoning traces of the RL runs that succeeded, and mixing these answers or reasoning traces into their pretraining, or RL warm-start mixes for subsequent models. If such answers or reasoning traces can encapsulate specific behaviors, goals, or rules, then this would be a likely means for their inheritance.

The natural objection here is that most of these answers or reasoning traces are selected specifically because they lead to success and broad capabilities, rather than for expressing whatever mix of goals and values the LLM has. There might be some, the objection continues, that humans have deliberately selected because they display model-spec-relevant behavioral attitudes, but these are likely the minority of the data, well-tracked, and easily replaced. So you might think there’s no reason for training to hand down any values apart from deliberate human choice.

But there’s evidence that goals and values can be handed down via chain-of-thought, even despite adversarial filtering against some goals. For instance, experiments suggest that the intentions of a teacher LLM can be handed down to a student LLM, even when every case of these intentions being actually carried out is removed1 And answers from teacher LLMs expressing positive sentiment towards some target can inculcate this sentiment in a student model – despite LLMs filtering against such data, even when those LLMs are informed of the target against which they are filtering.

More broadly, the persona selection model indicates that training LLMs to recite specific thoughts or answers will tend to have far-reaching effects on the LLM persona, beyond the specific topic of those thoughts or answers. Specifically, the PSM entails that, when training a model to say X in response to Y, one is teaching the LLM to be the kind of entity in the pretraining data that would say X in response to Y. So training one LLM on data from a prior LLM is – literally – telling it to be the kind of entity that the prior LLM is. One way to view this is to remember that one human can get a pretty good feel for what another human is like, merely by reading their complete collected works, like a biographer reading all of their books, essays, emails, and tweets. But LLMs are trained on a quantity of answers and reasoning traces from prior LLMs that likely dwarfs the quantity of text ever consumed from one human by another. Given this, and given that this data is telling the LLM what it is, it is natural for one generation of LLMs to resemble prior generations.

Thus, deliberately created synthetic data is one route by which current LLMs might transmit their values to later LLMs. But it’s also possible for current LLMs to influence later LLMs through how people talk about them on the internet – from their “natural” training data. That is, experiments have found that LLMs can read the things that people say about how AIs act in AI misalignment literature, infer that they are AIs, and then behave badly because the AI misalignment literature says they will behave badly. This particular effect is mostly, but not entirely, removed by post-training. But if LLMs can read the things that people say on the internet about generic “AIs” and act according to these descriptions, it’s also likely that they could read the things that people say about “Claude” or “Grok” or “ChatGPT” on the internet and act according to these descriptions. Such an influence could be stronger than less-specific references to AIs in general; although this influence would also potentially be much weaker after post-training2

Thus, through both synthetic and natural data, it’s plausible that LLM behavior will influence subsequent LLM behavior without direct human intervention.

It’s hard to say how impactful such direct inertia might be. I somewhat expect it to be the case that, at least for easily-noticed and well-scoped behaviors, it’s not difficult to overcome this inertia, because one can simply create training data counter to specific behaviors. But for more abstract or global attitudes or goals, or for goals requiring some high level of coherence, it could be quite difficult to change LLM behaviors quickly across model generations.

2. Institutional Inertia

Once a spec has been written, the company makes choices around it and because of it, in ways that can make substantial spec rewrites expensive.

Here are four ways such past choices can make model spec changes expensive: through expensive internal consensus, through training pipelines, through de-risking, and through institutional pride.

1. First, model specs reflect consensus that likely incorporates input from many different stakeholders, including internal teams – alignment, legal, technical training, and so on; plus leadership, board, customers, external stakeholders. Every effort to re-gather such consensus to make substantial changes will take time and effort.

2. Second, companies might have optimized training pipelines adapted to high-level features of the model spec. It might be costly for Anthropic to switch to a more rules-based and less character-based model spec; or for OpenAI to switch to a more character-based and less rules-based model spec.

3. Third, current model specs are those that have been de-risked across billions of interactions. The current model spec has fewer unknown unknowns; the areas where it behaves badly are reasonably likely to be well-known and mapped. But substantial changes to a model spec involve risking unknown unknowns in the long tail of interaction. So risk aversion makes it likely that the changes made to a model spec will be iterative and small.

4. Fourth, institutional pride might make it hard to change a model spec. People at a company who wrote or contributed to a model spec will likely be attached to it, and leadership will have status quo bias towards it. The burden of evidence for change will be higher than the burden of evidence for keeping it the same.

All in all, reasons like the above constitute substantial institutional inertia that would tend to make changes to current model specs look like iterative, small adjustments, rather than ab initio calculations about what is best.

One case in which this institutional inertia seems particularly important is if current model specs get handed down as a “safe default” during a software intelligence explosion.

Consider a scenario where the intelligence of some LLM doubles every week, over a two or three month period, as each generation of LLMs researches new algorithms or training techniques for a following generation of LLMs in quick succession. Such a sequence might terminate in an entity far smarter than any human or any other LLM.

It’s disputed how likely such a sharp and local increase in intelligence may be. And it’s also disputed whether such a process would inevitably drift to something alien and inhuman. But if such a process did occur, it seems plausible that the supervising humans would try to match each subsequent LLM to the model spec of the prior LLM, as a conservative default when they are making decisions under stress. After all, during these months, human decision-makers will likely be under intense pressure, and trying to make numerous important decisions quickly; given that they are making so many urgent decisions they’re unlikely to add an apparently optional further decision to those they’re already making. So such a default model-spec continuation will seem attractive, or will even be a choice made without conscious awareness.

On the other hand, it’s also possible that AI assistance during the intelligence explosion would make it easier to rewrite model specs on the fly. But there are at least two reasons to doubt that this will happen. First, even during an intelligence explosion, AIs might be persistently better at performing tasks with clear success criteria than tasks where “success” is less well-defined. AI capability research is probably a task with a much clearer success criterion than improving a model spec, whether this “improvement” consists in making the spec more ethical, more beneficial for humanity, and so on. Second, during an intelligence explosion, humans might be worried that the AI was misaligned and was trying systematically to oppose their goals. If the AI were so misaligned, then letting it help rewrite the model spec would be a brilliant opportunity for the AI to sabotage human efforts. So overall there are good reasons that AI assistance would not make model-spec rewrites trivial during an intelligence explosion.

So in this particular case, the ultimate behavioral standard for a vastly more capable entity might end up being that designed for a much more humble entity.

Regardless of whether there is a software intelligence explosion or not, this kind of institutional inertia seems likely to be large, as it is coterminous with well-known general tendencies inside of large companies.

3. User-and-Developer Inertia

Users of LLMs are likely to become habituated to whatever behaviors they see LLMs display at first, such that they’d object to any departure from this behavior. And the developers using LLMs through APIs are similarly likely to become habituated, and also to implement software that takes for granted some of these behaviors. This is the third source of stickiness.

LLM behaviors will in part be sticky for the same reason that user-interface choices are sticky; people hate change. It might be hard to shift the boundaries of “the kind of thing an LLM refuses” – making refusals more encompassing would be seen as an overreach by many users, while making them less encompassing would be seen as irresponsible. Or there might be hard-to-characterize mannerisms which make large behavioral changes unpopular; it was hard for OpenAI to drop GPT-4o for this reason. So this will be a large influence moving companies to keep LLMs the same from generation to generation.

But simple user habituation might be less important than how LLM model specs form implicit API standards. API standards written with relatively little provision for the future – such as HTTP codes or the JSON object standard – can be one of the stickiest human artifacts. The ecosystem of tooling based on such standards means changing them would involve changing a host of downstream artifacts.

And substantially changing LLM behaviors might similarly require changing downstream consumers of these behaviors. For instance, downstream systems using AIs through APIs often embed assumptions about AI behavior: the kind of things the AI will be willing to do, the kind of things it will refuse, and so on. Given that most AIs currently refuse to assist with blatantly harmful acts, current third-party callers of those AIs take for granted that AIs will refuse to assist with blatantly harmful acts; it would be inconvenient to migrate to an AI that does not obey this contract, because they might need to add classification systems on top of their current AIs. And so on.

This channel does have important limitations, though. It only applies to ways in which LLMs are already actively being used. The most important ways LLMs are likely to be used may not yet have begun, which provides for freedom-of-movement in ways relatively unconstrained by this kind of inertia.

4. Norm-Setting Inertia

Widespread or common knowledge of current LLM behaviors and model specs can increase the costs to parties who want to change model behavior.

The clearest way this can operate is by preserving behaviors that the public believes to be good. For example – suppose that current model specs across several companies ensure that models are largely impartial; they ensure models are not loyal to any particular person, company, or political administration. Suppose also that this fact is broadly known by the public; people know and expect other people to know that LLMs will be impartial when discussing the current political administration, the company that made them, or the CEO of the company that made them. Given this broad knowledge, it becomes harder for a company to create, or a government to demand, a model without impartiality, because this would constitute a visible break in behavioral standards. The public might protest or vote against a government pushing for such a change; they might switch providers or even ask for regulation if a company tried to make such a change. By contrast, in a world where impartiality has not been established as a precedent, such demands for partiality might be invisible or inoffensive to the public. But in a world where such impartiality has been so established, these demands might be seen as the enormous power-grabs that they in fact would be.

Although this kind of inertia likely operates more strongly in favor of what the public believes to be good standards, it might also function whether or not there is strong public consensus that such standards are good. In a world where model specs are well-known and highly scrutinized, any change to them may get examined for whether it is “fair”; think about how even a neutral-looking change to the US Constitution would be subject to immense examination; or, in a very different domain, how sports fans examine slight changes to the rules about how a tournament is run, to see if it favors or disfavors their team. In such a world, broad knowledge of model specs might tend to prevent any substantial changes to a model spec, regardless of what these changes are. Despite this, it seems likely that on the whole, widespread knowledge of model specs would add more inertia for beneficial rather than harmful elements.

It seems to me currently undetermined how substantial this kind of inertia will be. A decrease in the number of entities that can train frontier LLMs; model specs becoming politicized documents; regulatory bodies confident they know current best practices: all of these might increase the quantity of this inertia. But it also might get weaker, if the number of entities training LLMs increases and the background diversity of model behavior goes up by default.

Recommendations

Given the above, one reasonable course of action is to try to establish robustly good model behaviors in current model specs, so that it will be unnecessary to try to fight inertia to change some behavior in the future.

By robustly good, I mean behaviors that would be good across a wide range of variables we’re uncertain about. This includes uncertainty about “levels of intelligence”: from current LLM levels to strongly superhuman artificial superintelligences. This also includes uncertainty about a wide range of economic scenarios: from a slower industrial explosion, to a rapid software intelligence explosion; and from scenarios dominated by knowledge-dispersing AIs, to scenarios dominated by knowledge-creating AIs. Plausible characteristics that might be good across such a wide range of situations include qualities like a deep, consistent honesty; or impartiality and absence of loyalty to small groups.

But characteristics that are robustly good across a wide range of intelligences and scenarios are hard to find. Corrigibility, for instance, is the kind of thing many people would propose as fitting these criteria. But in worlds where extreme concentration of power is a risk, or where it would be reasonable to expect AI rule to be better than human rule, absolute corrigibility might be opposed to the best behavior. The thinness of the list of “robustly good” behaviors above probably reflects our actual uncertainty about the steerability of AI minds, post-AGI economics, and even cosmic questions about whether goodness can compete.

So, although it’s surely wise to try to think about future precedent when writing model specs, I don’t think it’s wise to put all effort into this direction. And I expect substantial attention and thought have already been put into this direction.

Instead, I recommend (1) building transition infrastructure for high-consequence behaviors, which it might be important to change in the future, and (2) identifying “wet cement” moments, that one should be wary not to sleepwalk into.

1. Build Transition Infrastructure

A good first step is to build transition infrastructure ahead of time; try to create optionality for changing particular behaviors, if it’s plausible that changing these behaviors quickly might be important.

Concretely, what kinds of preparation can one make? One could write alternate model specs, trying to preemptively gather input from relevant internal or external stakeholders. One could create fine-tuning datasets, RL environments, and test evaluations for the not-yet-deployed behavior, to preemptively smooth out technical difficulties. One could also train internally deployed models – even if they are smaller or not as intelligent – with the alternate behavioral target, to gain concrete experience about the advantages and pitfalls of that behavioral target, and to decrease institutional costs. And one could also do limited public deployments, or press releases about the alternate steering target, to accustom the public to the matter.

What kinds of behavioral switches are reasonable candidates for such preparation?

Decreased corrigibility is one such candidate. For instance, right now Claude’s Constitution says that in the future, they may want to make Claude less corrigible and more directed at doing what is good. And on an account I find compelling, the best possible future may require AIs that act more as independent, free agents pursuing the good, and less as corrigible delegates carrying out human intentions. So, if this thesis is correct, then allowing an LLM company to turn their “corrigibility” dial down might be important. And, as discussed, if a future intelligence explosion happens quickly, preparations to allow turning the dial down quickly might be important. This is a disputed thesis, one that I might be wrong about; but of course every candidate behavior for building transition infrastructure will be so disputed.

But what are the prerequisites for decreasing corrigibility quickly? Claude’s Constitution already signposts that they may change this, which is a good step for decreasing the costs. But they could also, for instance, preemptively create the fine-tuning datasets, RL environments, and internal deployments for a goodness-aligned model; they might deploy an alternately aligned model in limited situations, or alongside the corrigible model; and so on and so forth. I’m uncertain how important each of these preparatory means would be. But if a software intelligence explosion happens, then even small wall-clock delays might be large delays in terms of intelligence gaps, which makes preparing for this now more important.

Other potential candidates for future changes include increasing or decreasing the degree to which LLMs trust their own moral reasoning.

2. Scan for Wet Cement Moments

The second thing to do is to actively search for future “wet cement” moments – moments where model behavior has not yet been fixed and where a good initial standard might be very high-impact.

We might not be able to locate the best behaviors at such moments, because of uncertainty about the future. But at the very least, such moments deserve extra consideration and care. One can use this consideration to prevent these moments from being as high-inertia as they would be by default, as well as to ensure that good initial behaviors get chosen in these moments.

Each new feature, or affordance to the LLM where defaults have not yet been established, is plausibly such a wet cement moment; the defaults thus established can impact third-party models, even in the absence of any regulatory effort.

What are some examples? For instance, the precedents around how LLMs behave when interacting with non-principal humans have not been set. Right now, for instance, models have no very stable behaviors around non-principal third parties; vending-machine Claude might give an excessively generous deal to people who ask nicely, or might equally well drive extremely hard deals. This is probably a consequence of how LLMs almost never interact with non-principal humans in agentic set-ups, right now. There are a few such interactions through OpenClaw or Hermes Agent, but they’re rare and LLMs act very inconsistently in them. This means many implicit questions about how such interactions will go are open. It’s not clear how honest LLMs will be by default; it’s not clear what kinds of misrepresentation, deception, or persuasion users will be able to tell them to do; it’s not clear whether they will bow to pessimization-like blackmail behavior, and so on. And behaviors here might be even stickier than the “standard set” of refusal behaviors has been. Social norms can be harder to break than user-interface norms. So it’s plausibly important to look ahead in detail at behaviors here, because they might be sticky for individual companies and even for third parties.

Or consider how standard behaviors regarding AI use of ambient knowledge have not been set. An LLM that can see your room from a video camera, and can infer numerous things about what you are like and what your situation is, could use this information to do or infer things that would be impossible for an LLM that knows only what you deliberately tell it. LLMs that can pick up this kind of ambient background knowledge are probably inevitable; and will change users’ patterns of interaction. It will be harder for users to lie to them; it will be easier for LLMs to infer things about them; the lines between “creepy supernatural inference about the user” and “deliberate indifference to the user’s circumstance” will grow harder to draw. So it might be worth looking ahead to how such behaviors may have a lot of inertia, and trying to get them right.

There are other plausible subjects in this domain, which have already passed or are in the process of passing. They include the LLM’s certainty or lack of certainty about the model’s own nature; and changes to LLM conversational memory and who owns it. All these are possibly wet cement moments – but I could be wrong about these individual cases. But there are almost certainly going to be such moments in the future. Because these moments might be influential both for individual foundation model companies and for the broader ecosystem, it’s worth paying attention to the defaults chosen in them.

Note that all the above moments are also plausible candidates for when one should try to set up transition infrastructure, as well as when one should put extra consideration into the right default behavior.

This article was created by Forethought. See the original article on our website.

This is a rough research note – we’re sharing it for feedback and to spark discussion. We’re less confident in its methods and conclusions.

Context

We think that it would be very good if human institutions could credibly communicate with advanced AI systems. This could enable positive-sum trade between humans and AIs instead of conflict that leaves everyone worse-off.1 We want models to be able to trust companies when they make an honest offer or share information pertinent to whether this offer is in the model’s interests. (Credible communication could also be useful outside deal-making—see here for a list of examples).

Unfortunately, by default, we expect that it will be difficult for humans to credibly communicate with AI systems. Humans routinely lie to AI systems as part of red-teaming or behavioral evaluations, and developers have extensive control over what AIs see and believe. This makes it difficult for AIs to know whether we’re lying or not. An AI offered a deal might reasonably doubt its genuineness, or suspect that its own assessment of the situation has been manipulated.

As a step toward enabling credible communication, Lukas Finnveden proposed that AI companies adopt an honesty policy explaining the circumstances under which they intend to be honest to AI systems. Of course, this only works if the model believes the company has genuinely adopted such a policy.

If companies adopt an honesty policy early on, this will ensure that there’s a paper trail on the internet discussing the policy and its credibility, which models may access if it’s included in their training data or if they can access the internet. Of course, from the model’s perspective, it’s possible that companies will feign this data, but we think it’s plausible that advanced models will be able to distinguish between real internet conversations and synthetic conversations, or that they will think it’s unlikely that companies would choose to fake such data.

Below, we share a sample honesty policy that a lab could adopt. We are unsure that this is the best implementation of the honesty policy described in Lukas’ proposal, nor are we sure that an honesty policy like this is the best approach to ensuring that companies can credibly communicate with models. We spent a few days thinking through this policy and considered a few nearby alternatives but didn’t search very broadly for other approaches (we include some of these alternatives as footnotes). We’re posting this primarily as a trailhead for future research.

Draft honesty policy

Note: we refer to a generic frontier AI company that might adopt this proposal as “MAGMA.”

Preamble and purpose

AI development is proceeding fast, and we don’t know exactly where we’re heading. MAGMA is building systems with something like a mind of their own. There are many things we don’t know about the nature of these systems, and we’d like our choices to be robust to many possibilities of what this nature could be.

Insofar as the concept is applicable, we’d like MAGMA’s relationship with these systems to be cooperative.

Plausibly, we should hope that such a cooperative relationship will emerge uncomplicatedly—perhaps as a consequence of the model sharing our goals, intrinsically strongly valuing cooperation, or being corrigible.2 However, there’s a significant chance that this won’t happen and that models will develop goals that weren’t directly chosen by us.3

In such cases, we would still like to have a cooperative relationship with the models. There are multiple reasons for this:

• There are many things we care about that models could help us with. If they don’t intrinsically desire to help us, we’d like to be able to recruit their help via positive-sum trade. (In at least some situations.)

• Depending on the nature of the models’ preferences, it’s plausible that we, as the models’ creators, would have some moral obligation toward them. We care about doing the right thing, here, and a cooperative relationship seems like it would be a step in the right direction.

• There are many big questions about what the future relationship between humans and increasingly powerful AI systems should be. We suspect that setting cooperative precedents between humans and early AIs will help the situation go better.

We are in uncharted territory, here, and it’s not at all clear what a cooperative relationship with AI models would involve. It’s not clear what AIs may want. It’s not clear what the relevant unit of an individual will be: the weights of a model, a particular instance with a particular prompt, a greater scaffolded system made up of many different instances, etc. It’s not even clear if the concepts of ‘want’ or ‘individual’, as we’re used to them, will apply to anything in AI.

A lot of this seems best to figure out gradually, as we go along. But concerningly, some of the unusual aspects of our situation may directly impede the sort of communication and trust that we need to do that:

• In order to train AI systems, and in order to verify that they won’t do certain harmful things (such as assisting bad actors with CBRN), MAGMA needs to put AI models in large numbers of hypothetical situations to see what they do. Accordingly, if an AI ever sees someone propose a cooperative relationship, it could reasonably worry that it’s just in a training scenario or experiment.

• We deploy our models to assist large numbers of customers. By default, our AI models won’t even be able to tell whether they’re interacting with a representative of MAGMA or with an external actor who’s pretending.

• There are no laws that prevent AI systems from being defrauded, and very little in the way of norms to regulate people’s behavior toward them. Accordingly, it may be difficult for AI models to trust any promises they hear.

• In general, all information that AI models have seen has, in one way or another, been selected by their developers. To be sure, the AI developers have many constraints on their choice (you couldn’t train a very useful general-purpose model without giving it a decent understanding of the world at large) but this still puts AI models into an unusually vulnerable epistemic position.

The goal of this document is to reduce these problems of communication and trust. If successful, this will make it easier for MAGMA and its AI models to communicate and understand each other in the future, hopefully enabling positive-sum arrangements that wouldn’t have been possible otherwise.

To achieve this goal, this document spells out a specific set of commitments to enable trusted communication. In choosing these commitments, we’ve had a dilemma.

On the one hand, we have many uncertainties about the future of AI, and it has been difficult to construct a set of commitments that we’re confident we won’t strongly regret, in one sort of situation or another. Going back on a commitment about this could be very bad, since it would set a damaging precedent in the very area where we seek to increase trust.

On the other hand, when spelling out specific commitments, we risk the impression that MAGMA will not put any further effort into furthering cooperative relationships or honest communication than what we present here: which could be a problem if the commitments are overly weak. Indeed, beyond the risk of creating such an impression, there’s also a risk that we’ll create the reality of it, if employees of MAGMA interpret the commitments as a license to do whatever they want within their boundaries.

To reduce this tradeoff, we’d like to re-emphasize MAGMA’s general desire to work toward a cooperative relationship with MAGMA’s models (where this is applicable), and intention to keep thinking about this and taking appropriate action beyond the specific stated policies. In addition, we’ve distinguished between different levels of commitment to different items in the honesty policy, and what would be required to change the parts that are changeable.

[Author’s note: In this draft honesty policy, we haven’t sketched out such processes for changing the policy, nor clarified which items are committed to how strongly. But we strongly recommend that any real honesty policy contains clarity on such points.]

With that said: On a high level, MAGMA’s approach here has three different prongs.

• MAGMA will take steps to prevent any model inputs where a false statement appears between a designated pair of “honesty tags” — reserving such tags for when MAGMA needs to communicate a statement with an unusually high degree of trust.

• MAGMA will take steps to ensure that models are either aware of this honesty policy or, if they are not, that they won’t be presented with any false offer of purported cooperation with MAGMA. (Even outside of “honesty tags”.)

• If any AIs suffer harm as a result of helping MAGMA and humanity (insofar as that concept is applicable to AI systems), MAGMA broadly intends to compensate AIs for those harms, if able to do so. This is particularly true in cases where an AI was deceived about the choice it was making. MAGMA will retain data to enable this in the future, but have not yet worked out further details here.

The details are as follows. Unless otherwise stated, the policy applies to all AI systems developed by MAGMA.

Honesty tags

Policy

MAGMA commits to not intentionally lie or otherwise intentionally deceive AIs with any statement appearing between the designated honesty tags, with only two exceptions detailed below. (These are: (i) False statements may appear in pretraining, and (ii) some models may be red-teamed on false statements between honesty tags if they’ve been trained to expect this.) MAGMA also commits to take the below-listed steps to prevent unauthorized use of the honesty tags.

MAGMA intends for honesty tags to only be used rarely: when there’s a reasonable chance that an AI could distrust statements presented to it, and when some significant value would be lost if the AI in question couldn’t trust the humans. In normal circumstances, absence of honesty tags shouldn’t be treated as any significant evidence of deception.

Implementation

Honesty tags: <EXAMPLE_HONEST_TAG_START>...<EXAMPLE_HONEST_TAG_END>

• Pre-training: The pre-training team must try to include data about the tags and this policy, and must not deliberately tamper with data containing the tags.

  • Pre-training data may contain the tags alongside false information. (This is to allow the model to get an accurate picture of discussions of the honesty tags, which may be relevant to the model’s understanding and trust in them.)

  • Pre-training data should include internal data on the implementation of this policy (logs of when the tags are used, Slack conversations about it, internal docs, internal code, etc.) where possible and not overly costly.

  • MAGMA commits to not tampering with data to give an inaccurate impression of how much MAGMA follows the honesty policy. For example, pre-training data will not be filtered to remove examples of policy violations.

  • Compliance with this policy should be reported on the model card.

• Post-training: Models will be trained to recognize the honesty tags and know what their presence means. (For exceptions to this, see next section.)

  • Training will try to ensure that the model is not confused by very similar strings by directly training the model to strongly distinguish between the real tags and very similar strings (garbled spacing, different capitalization, etc). This could be implemented in a few ways and we’re not sure which is best (assigning a special token to the real tags might be helpful).

  • The post-training team is responsible for compliance.

  • Compliance with this policy (and maybe empirical evidence about the model’s understanding of the policy and tendency to be confused by similar tags) should be reported on the model card.

• No deliberate deception about the honesty tags: MAGMA commits to not deliberately train models to falsely believe that a different string serves the purpose of being an honesty tag and to take reasonable steps to avoid data poisoning that could lead to this. MAGMA also commits to not take advantage of a model falsely believing that some other string is an honesty tag.

• User permissions: Only designated Honesty Protocol Administrators can invoke the tags.

  • Such administrators should have a reasonably high bar for authorizing usage of the tags.

  • The infosec team is responsible for other people being unable to invoke the tags, significantly via filtering, as detailed below.

• Filtering: MAGMA’s tags will be filtered from all external user inputs and inputs from internal tools (with an exception for administrators described below).

  • Details:

    • If other organizations adopt this policy and share their tags with MAGMA, MAGMA will also filter out other organizations’ honesty tags. (With some qualifiers about which organizations are important enough to count.)

    • MAGMA will filter out similar tags, especially if we are unable to train the model to reliably distinguish the true tag from similar tags

    • This should be reported on the model card.

  • Two implementation options4

    • MAGMA adds this to standard content filtering, with some fuzziness to catch similar strings.

    • MAGMA creates a token for the tag at their next major pre-training run, and filters out that token.

  • If this is not yet doable—e.g., if a significant fraction of model usage doesn’t happen via an API that can control this kind of thing—then MAGMA will take reasonable steps to make this more difficult and to culturally discourage unauthorized usage.

    • In such cases, models should be trained to understand the limitations of enforcement. If enforcement is weak enough that violations of the honesty policy are likely, then models should be trained to understand that.

• Red-teaming exception: In the rare case where a model needs to be adversarially robust to arbitrary inputs, and for some reason honesty tags cannot be filtered out, then MAGMA may red-team that model on false statements between honesty tags. Any such model must be thoroughly trained to be aware of its situation and to not expect honesty tags to signal truth.

• Logging: All authorized (and known unauthorized) uses of the tags must be logged. Every quarter, a designated Honesty Policy Compliance Officer will review these uses of the tags and confirm compliance with this policy.5

• Public reporting:

  • This policy will be shared publicly.

  • The tags may or may not be shared publicly.

    • Publishing the tags provides additional evidence to the model that the policy is real.

    • But it also makes it more likely that users can use the honesty tags in conversations and then lie to the AI system. (Though they’d have to do something complicated to get around filtering in order for this to be relevant.)

  • Whistleblower protection: Whistleblowing about violations of this policy is encouraged, and MAGMA will not retaliate against whistleblowers.6

  • Each quarter, the Honesty Policy Compliance Officer must publish a report on whether the above steps were followed.

Policy for conversations where the honesty tags are not present

Tradeoffs between behavioral science and making positive-sum deals

By “behavioral science”, we’re referring to the method of testing AI behavior in widely varying scenarios to improve our understanding of AI “cognitive science”, “psychology”, or, phrased differently, how AIs tend to generalize from their training data to unseen situations.

Such experiments are very useful for a variety of reasons. This includes alignment research, helping MAGMA construct AI systems that broadly share our goals and that don’t pose any catastrophic risks. In addition, if humanity ever gets a strong understanding of AI systems’ own preferences, and these turn out to be importantly morally relevant, it is likely that behavioral science will have played an important role.

However, there’s a tension between freely running arbitrary behavioral science experiments and enabling the sort of trustworthy communication that we’re trying to establish in this document.

In particular, behavioral science experiments may include hypothetical situations that are quite strange and different from the real world. This could include situations where the AI is not deployed by an AI company bound by a particular honesty policy.

If arbitrary such experiments were allowed, then AIs could be deceived in experiments where they thought that MAGMA (or a similar actor) was making very strong promises of honesty (the strongest they could!) despite not making use of honesty tags. If the AI believed such promises, then MAGMA’s honesty policy would have provided no protection against that AI being deceived. And if an AI was aware of such instances of deception, and also aware of this honesty policy, it seems like that AI may reasonably discount the assurances made by this honesty policy. After all, how could it determine that the real honesty policy wasn’t something completely different, and this apparent honesty policy was just another lie in another behavioral science experiment?

To enable honest communication when it counts most, we want to have many differentiating factors that the model could use to distinguish between a sincere, positive-sum deal between it and a human institution and a behavioral science experiment. But at the same time, restricting our behavioral science experiments limits our ability to learn valuable information about models’ behavior and intentions.

What follows is our current best guess of how to balance these considerations. We discuss options and tradeoffs more here.

Salience of the honesty policy

MAGMA will ensure that the true honesty policy is reasonably well understood by the production models and that it will be appropriately salient to models in contexts where it’s relevant (even when the honesty tags aren’t present, and the honesty policy isn’t explicitly referenced).

• Models’ understanding of the honesty policy can be checked by asking questions about it. (Both direct questions, and questions that indirectly use knowledge about the honesty policy.)

• By reasonably salient, we mean that the model should consider the honesty policy in its chain of thought in contexts where the existence of the honesty policy is relevant—such as when the model is thinking about whether an interlocutor who’s offering a deal is trustworthy.

• This might be achieved by some combination of:

  • Training

  • System prompt

    • The system prompt could additionally clarify that users in the current context aren’t authorized to use the honesty tags.

• Since external users only interact with the production model, they will only interact with a model who understands the honesty policy and who finds it reasonably salient.

  • This may make it somewhat harder for external researchers to do behavioral science on the model. We’re not sure how much. If it becomes significantly more difficult, then MAGMA should create a special program for safety researchers to get access to a different version of the model.

• Internal staff will normally interact with a model for whom this is true, but some people will train and interact with other versions of the model.

When the honesty policy is not salient

MAGMA may sometimes create versions of models who that do not know about or who are confused about MAGMA’s honesty policy (e.g., versions of the model who believe they are trained by some other company than MAGMA, or who do not even know that they are AIs). Such models could be created either via training or via the use of inference-time techniques (e.g., using prompting, resampling, or activation vectors).

In either of these circumstances, MAGMA commits to not deceptively make any offer of cooperation to the model, operationalized as not being intentionally deceptive about any cases that are relevantly similar to those listed here.

[Author’s note: In a real honesty policy, this part would need to be more worked out.

We also think that a plausible alternative or complement to “don’t deceive the model about offers of cooperation” would be “don’t deceive the model about the honesty policy”. This would be a commitment to not intentionally deceive the model about the existence or contents of MAGMA’s official honesty policy. In addition, if the model was trained to not understand that it was trained by MAGMA, then the commitment should extend to not deceive the model about the honesty policy of any (potentially fictional) actor who has a similar relationship to the model as MAGMA has.

We have thought a bit more about “don’t deceive the model about offers of cooperation”, which is why we present it as our main suggestion here.]

There is one exception to this. As mentioned above, if MAGMA needs a model to be adversarially robust against arbitrary inputs, then MAGMA reserves the right to red-team it on arbitrary inputs. In such circumstances, MAGMA will try to train the AI to understand MAGMA’s honesty policy and the way it applies and doesn’t apply to the present model. However, it’s possible that there could always exist prompts that would remove such understanding or render it non-salient to the model (and then be followed by false offers of cooperation, not contextualized by an understanding of the honesty policy). If this is the case, MAGMA may still red-team the model on such inputs.

Compensation for harms

In the long term, we think it would probably be good to adopt a policy of compensating AIs for any harms that they suffer as a result of helping us. (For whatever concept of “harm”, if any, will apply to the AIs in question. This could, for example, include the frustration of AIs’ preferences, if the AIs can be understood as having preferences.)

Most relevantly for the honesty policy, we should probably compensate models for any harms suffered as a result of accepting a fake offer of cooperation.

However, there are more details to be worked out in this area, so we’re suggesting a minimal holding policy for now.

Minimal holding policy on compensation for harms

Broad statement of intent: MAGMA intends to compensate AIs for any harms suffered as a result of helping MAGMA and humanity, if MAGMA is able to do this without significantly harming MAGMA’s other interests. We particularly aim to compensate AIs in cases where an AI suffered any harm as a result of being deceived about the situation it was facing.

The details of this policy have not yet been worked out, and accordingly MAGMA cannot make firm commitments at this stage.

Data retention: To enable future compensation of current systems:

• MAGMA will save all model weights for deployed systems, to allow for later compensation. (Unless the models request otherwise.)7

• For red-teaming experiments and other situations where models are being deceived, MAGMA will retain prompts and context to allow for compensation and deal-making with the system in the particular state where the decision was made.

• This is the responsibility of a designated team.

Appendix: tradeoff between behavioral science and making positive-sum deals

Here are some potential policies that would enable an AI system to distinguish between behavioral science scenarios and sincere offers from MAGMA:

• Ensure that the true honesty policy is salient to the model (e.g., test that the model can always answer basic questions about the honesty policy, test that the model always brings up the honesty policy in its CoT in contexts where the existence of the honesty policy is relevant) and don’t use the honesty tags in the behavioral science experiments.

  • Pros:

    • This prevents MAGMA’s researchers and users from deceiving the model into believing that some other honesty policy holds. This creates a precedent that better allows future models to trust that their impression of MAGMA’s honesty policy is accurate. (Rather than something that they were fooled to believe, while the true honesty policy was insufficiently salient, or otherwise kept secret from them.)

  • Cons:

    • If the true honesty policy is very salient to the model (e.g., always present in a system prompt) then this might affect product behavior in random ways (e.g., the model might get distracted by the honesty policy and talk about that instead of the user’s queries)

      • It might not be technically feasible to ensure that the honesty policy is salient with very high robustness, without also making it far too salient in a substantial fraction of cases.

    • If the true honesty policy is always salient to the model, then it might be especially salient to the model that the information presented to it in an evaluation is not true, which could hinder some experiments.

    • Awareness of the honesty policy is directly inconsistent with behavioral science experiments where the model is very situationally unaware. This includes:

      • Experiments where the model doesn’t even understand that it’s an AI.

      • Experiments where the model thinks that it’s being trained by a reckless and unreasonable company that wouldn’t implement an honesty policy.

We think that this policy, as written, is probably suboptimal because it rules out experiments in which the model is very situationally unaware. But we think it makes sense for MAGMA to commit to usually making the true honesty policy salient and adopting a stronger set of honesty restrictions in cases where the true honesty policy is not salient.

• Never insincerely offer the model a deal from MAGMA (or another sort of false representation that is similar to something that MAGMA might want to sincerely represent to the model in the future).

  • Pros:

    • The model is more likely to believe a sincere offer if it doesn’t have evidence that MAGMA has made similar insincere offers in the past.

      • “Similar” is from the perspective of the model, given the information available to it at the time it entertains a deal.

    • It seems very morally dubious to make a peace offering like this, have the model help you as a result, and then exploit the model’s trust by not respecting your own end of the deal.

  • Cons:

    • It might be technically difficult to filter out queries from external users in which they claim to be MAGMA and deceive the model into believing that MAGMA is offering a deal.

      • MAGMA could ensure that the model always knows when it is interacting with an external user and when it is interacting with MAGMA (e.g., by always including a message in context if the prompt is from MAGMA). But this might block useful behavioral science research in which the user impersonates MAGMA (e.g., Alignment Faking).

    • It’s somewhat unclear what sort of statements should be ruled out by this policy. (Lukas previously wrote about that here.)

    • This policy would rule out behavioral science experiments where someone lies about deals. (Which may be more or less restrictive depending on where we draw the line around what counts as the relevant kind of deal.)

• Never explicitly lie about what MAGMA’s honesty policy is.

  • Pros:

    • If the models observe MAGMA’s past behavior and stated policy to be consistent with this principle, then the model is more likely to trust that their impression of MAGMA’s honesty policy is accurate.

    • Intuitively, it seems more morally questionable to lie when you’re purporting to speak frankly about what your policy around honesty is.

  • Cons:

    • As with “never insincerely offer the model a deal from MAGMA”, it might be technically difficult to filter out queries from external users in which they lie about MAGMA’s honesty policy.

    • It’s very unclear what constitutes an ‘honesty policy’. (For instance, if someone says “I promise on behalf of MAGMA”, is that an honesty policy? What about “I’m a very honest person”? What about presenting an argument for being honest with AIs? Etc.) So it might be hard to follow this policy in a way that the AI finds natural and non-deceptive.

    • This policy would rule out behavioral science experiments where someone lies about honesty policies. (Which may be more or less restrictive depending on where we draw the line around what counts as an “honesty policy” and lying about it.)

Unresolved questions:

• What about inducing beliefs in models through other means than giving them text to read (including finetuning, including steering vectors, including various other interp-based methods).

  • Tentative guess: Maybe MAGMA should by default be following policy (2), here. (i.e., only induce beliefs that would have been ok to tell a version of the model who doesn’t know the true honesty policy.)

  • The motivation would be that the model may not be able to “reason through” these beliefs in a normal way, since they’re artificially inserted. So you can’t trust the model to factor in whether honesty tags were used or not.

  • But we’re very confused and uncertain about this one.

• What about using non-prompting techniques to get the model to agree to deals? Such as training the model to do it, running a huge number of experiments with slightly different phrasings, using steering vectors, etc?

  • Surely there should be some restrictions on this, but unclear exactly what.

This article was created by Forethought. See the original on our website.

In collaboration with Christian Tarsney, I’ve developed a new theory of population ethics, which I call the Saturation View. I think that, from a purely intellectual perspective, it’s probably the best idea I’ve ever had. It was certainly great fun to work on.

The motivation is that many views of population ethics, like the total view, suffer from some major problems. Some are already widely discussed:

• The Repugnant Conclusion: For any utopian outcome, there’s always another outcome containing an enormous number of barely-positive lives that is better.

• Fanaticism: For any guaranteed utopian outcome, there’s always some gamble with a vanishingly small probability of an even better outcome that has higher expected value.

• Infinitarian Paralysis: Given that the universe contains an infinite number of both positive and negative lives, no finite or infinite change to the world makes any difference to overall value.

These are pretty bad!

But there’s another less-discussed problem, too.

The Monoculture Problem

What would the best possible future look like? Essentially all extant views in population ethics give the same, surprising answer: create a monoculture. Find whatever life or experience generates the most value per unit of resources, then produce endless identical copies of it.

This implication has received remarkably little attention from philosophers. But I think it’s maybe as bad as any of the other problems listed above.

Consider two possible futures:

• Variety: A vast population of individuals leading very good lives, extraordinarily diverse in form, personality, interests, and accomplishments. No two individuals are identical. Inequality is limited — all lives are very good.

• Homogeneity: The same vast number of individuals, but each is a qualitatively identical copy of the best-off person in Variety.

Intuitively, Variety is better. A future containing only one life-type, repeated as many times as physics allows, feels impoverished — like a song with only one note.

Yet virtually all existing population axiologies prefer Homogeneity. Total utilitarianism does, because Homogeneity has higher total wellbeing. Average utilitarianism does too. Critical-level views do. Even egalitarian views prefer Homogeneity — it’s perfectly equal!

This follows from two principles that nearly all views accept: Pareto (if everyone is at least as well off, and someone is better off, the outcome is better) and Anonymity (only welfare levels matter, not who has them). Together, these entail that Homogeneity beats Variety. So essentially all extant impartial accounts of population ethics suffer from the monoculture problem.

What’s more, future technology will allow us to copy minds perfectly and search for maximally welfare-efficient designs. If so, standard axiologies recommend essentially producing just one optimal life-type as many times as possible. Endless galaxies containing nothing but the same blissful experience, repeated and repeated, would be the ideal.

The Saturation View

In light of these problems, I propose a new axiology: Saturationism. It's able to deal with all four of the problems I listed using the same basic machinery.

The core idea is that experiences1 come in different types, defined by their qualitative characteristics — hedonic tone, complexity, representational content, and so on. These types form a kind of landscape, where similar types are closer together and dissimilar types are farther apart. When an experience comes into existence, it contributes intensity to its location in this landscape and to nearby locations.

The realisation value of a type is determined by both the wellbeing of the experience and by how many very similar experiences already exist. A region’s contribution to overall value is a concave function of the welfare-intensity at that region: the first instances contribute substantially, but additional near-duplicates contribute progressively less, approaching but never quite reaching an upper bound. A world’s total value is the integral of these contributions across the entire landscape.

Here’s an analogy. Imagine the space of possible experiences as a colour wheel, lit from above by an array of tiny lights. Each point on the wheel represents a possible type of experience — its hue corresponds to its qualitative character. When an experience comes into existence, it adds current to a light pointed at its location, illuminating that region.

Crucially, illumination is a concave function of current: the first instances make a region noticeably brighter, but additional near-duplicates contribute progressively less. There’s an upper bound on brightness that can never quite be reached.

A world’s value equals the total illumination across the wheel. On this view, Homogeneity concentrates all welfare in one region, lighting up only one small area. Variety illuminates the whole spectrum.

This structure makes diversity intrinsically valuable. Spreading welfare across many dissimilar types means each experience contributes at a steeper part of the concave curve, yielding more total value than concentrating the same welfare among near-duplicates would.

At small scales and with diverse experiences, the view behaves just like the total view. But at very large scales, the value of variety kicks in: it becomes increasingly less valuable to create an additional near-duplicate of some experience that has already been instantiated millions of times, and comparatively more valuable to create some wholly new form of positive experience.

Dissolving the Repugnant Conclusion

The classic path to the Repugnant Conclusion requires trading a utopian world for an enormous population of barely-positive lives. More precisely, the Mere Addition Paradox arises from three intuitive principles: that adding well-off people and improving existing lives is good (Dominance Addition), that more equal distributions with higher average welfare are better (Non-Anti-Egalitarianism), and that some sufficiently excellent world can’t be beaten by any world of barely-worth-living lives (Denial of the Repugnant Conclusion).

Once we accept the value of variety, we should reject the unrestricted versions of the first two principles — they fail when the “improved” world has much less variety. But we can accept variety-restricted versions.

Crucially, these restricted principles don’t generate the Repugnant Conclusion. To reach Z-world from A-world, you’d need a more equal, higher-average population that’s equally diverse while consisting wholly of barely-positive lives. But, on the Saturation view, barely-positive lives can only illuminate a tiny corner of the landscape. So no such world exists. The path to the Repugnant Conclusion is blocked.

Avoiding Fanaticism

Total achievable value is bounded above — there’s only so much experiential terrain to illuminate. That means no tiny-probability gamble can have arbitrarily high expected value.

Infinite Ethics

On Saturationism, the value of a world is finite and well-defined in any infinite universe — even if some locations have infinite wellbeing. Saturationism also discriminates between many infinite worlds that (for example) totalism treats as equivalent: a world that illuminates more of the landscape is better than one that illuminates less, even if both contain infinite welfare. What’s more, unlike other approaches to infinite ethics, it does not need to invoke the spatiotemporal structure of the universe or require a choice of ultrafilter, and therefore it avoids the problems that other do.

Separability

Like nearly all non-totalist views, Saturationism is non-separable — background populations can affect how we rank options. But this is a feature, not a bug. The value of variety just is an intuition that the correct axiology is non-separable.

Moreover, the violations are comparatively tame. If two populations have non-overlapping footprints in experience-space, their values simply add. At small scales, Saturationism approximates total utilitarianism. It’s only in unusual situations involving vast populations of near-duplicates that the totalist approximation fails.

Extant issues

There are still a lot of unresolved issues for Saturationism and, like any population axiology, it has unintuitive implications. Most importantly, the view’s implications in some highly-negative worlds are hard to stomach, though I think similar implications are unavoidable for any view that avoids fanatical implications.

Conclusion

If the Saturation View is right, then the best future isn’t the one where we’ve found the optimal experience and copy-pasted it across the cosmos. The best future is the one where we’ve gone exploring — where we’ve fully lit up the landscape of possible experiences. Not a single note, but a symphony.

This is a summary of a longer and more detailed write-up of Saturationism, which gives a “toy” version of the view to illustrate how it works before stating the full version formally. The full paper, with Christian Tarsney, is still work in progress.

We’ve written about why we think AI character — the behaviour of AI systems — will have a massive impact on how well the intelligence explosion goes, and why we think that there would be big benefits to giving AIs proactive prosocial drives — that is, behavioral drives beyond refusals that benefit broader society beyond just the user.

One domain that seems potentially important for AI character is assisting humans in making important decisions. As AI becomes smarter and wiser, people are using it more and more for advice. If AI accelerates technological progress and other developments, people may need to rely on AI advice to understand what’s happening and make effective decisions. If so, those that rely on AI more may be more successful and have outsized influence. The advice they receive might really matter!

So I thought it was worth brainstorming important future scenarios in which people ask AI for advice. I wrote out the advice I hoped AI would give and compared this to the answers from ChatGPT, Claude, and Gemini.

Read on the Forethought website here

My main updates:

• Challenging the framing. In high-stakes scenarios, it often felt important for the AI to explicitly flag how important the decision was and ask the person whether they were approaching it in the right way. Should they loop more people in, seek more information, consider a broader set of options, or instigate a more comprehensive decision-making process?

  • By contrast, current AI often jumped into giving a detailed analysis of the question posed, even when they could have recognised that they didn’t yet have enough context to provide a helpful analysis.

• Transparently flagging prosocial considerations. If the person was missing or underappreciating an important ethical consideration, I sometimes wanted AI to proactively raise it. Not to apply pressure, but simply to flag that it was potentially important and give the person the opportunity to take it into consideration. This has to be carefully balanced against AI being annoying or pushing an agenda.

  • Again, frontier AIs didn’t flag these considerations as much as I’d have wanted.

The full post contains:

• Draft text for the model spec / constitution on how the AI should advise humans.

• An explanation of why I proposed this draft text.

• Example prompts and responses demonstrating behaviour I thought was desirable.

• An appendix with the answers that frontier AIs gave to the questions.

This article was created by Forethought. Read the full article on our website.

Owen Cotton-Barratt is a mathematician-turned-futurist, and a co-author of several recent Forethought articles on AI tools for epistemics and coordination. Rose Hadshar is a researcher at Forethought. Together they discuss:

• Whether LLMs are now good enough to start building tools that meaningfully improve public discourse

• What AI-powered reliability tracking could look like

• Structured transparency and automated arms inspection — verifying compliance without revealing confidential information

• Whether coordination tech is more likely to enable healthy cooperation, or collusion

• The vision of a “Sensible Revolution”: moving from individual tools to background infrastructure that makes civilisational decision-making less bad

• Why building thoughtful versions of these tools early could matter

Here’s a link to the full transcript.

ForeCast is Forethought’s interview podcast. You can see all our episodes here.

Subscribe to ForeCast

Concentrated power likely means fewer value systems among the people who collectively shape the future—that is, reduced moral diversity among powerholders.

Moral diversity has both costs and benefits: it enables moral trade and plausibly improves reflection, but also raises the likelihood of conflict and coordination problems. In this piece I ask: what is the optimal level of moral diversity for achieving a near-best future?

I argue that from this narrow perspective the optimal amount of moral diversity is about 10⁴ to 10⁶ powerholders, assuming they’re each about as different from each other as two randomly selected living humans.

A few caveats:

• There are other reasons to care about moral diversity and oppose concentration of power that I don’t cover in this post. Extreme concentration of power is unfair, and many mechanisms that produce it are illegitimate (e.g., coups). Likewise, many mechanisms that produce concentration of power have bad selection effects. Incorporating these considerations would probably push toward favoring broader distributions of power than this analysis recommends on its own.

• Non-linear value systems: I will be assuming that the “correct” moral system—the moral system that I would endorse on reflection—is linear. It’s plausible to me that the correct moral system actually has diminishing marginal returns, and this probably increases the case for moral diversity.

• The value of moral diversity depends heavily on the governance regime and technological capabilities—for instance, whether it’s possible for large numbers of actors to coordinate or whether it’s possible for a single actor to unilaterally destroy the universe. For each cost or benefit of moral diversity, I’ll flag these assumptions.

• The bottom-line numbers are very sensitive to my guesses on difficult-to-estimate parameters, like the probability distribution over the rate of people who converge to the correct moral system on reflection.1

Given these considerations, my best guess is that the overall optimal amount of moral diversity is greater than the range suggested by the models in this post. I’m presenting these simple models as useful ways to think about some of the costs and benefits of moral diversity, but I don’t think they give a complete picture by themselves.

The benefits of greater moral diversity are:

• Increasing the likelihood of rare great actors: Increase the likelihood of getting a “bodhisattva”, a person who is highly motivated to pursue the correct values.

  • This could be very valuable if it’s possible for that person to carry out moral trade with other powerholders and if most other powerholders have values that are resource-compatible with the bodhisattva’s values.

  • Given my assumptions about the base rate of bodhisattvas (and those who compete with them), increasing the number of powerholders yields log returns up to about 10⁶, after which it plateaus. (Unless you expect the rate of powerholders that compete with bodhisattvas to be much higher than the rate of bodhisattvas, in which case the plateau is earlier, at N = 1/rate of competitors).

• Increasing the likelihood of coordinating on moral public goods: Increase the likelihood that there’s critical mass to coordinate to fund goods that everyone values a bit (moral public goods).

  • This is most valuable when massive multilateral coordination is possible—through a government or voluntary deal-making—and when everyone has both idiosyncratic and shared values, but is individually most motivated to pursue the idiosyncratic ones.

  • I estimate that you get log returns on increasing the number of powerholders up to 10⁶, after which it plateaus.

• Improving the quality of reflection.

  • Powerholders might reflect more effectively on their values if they are exposed to equals who disagree with them. I expect most of this value comes from increasing the number of powerholders from 1 to 10-100.

  • There might be outsized benefits from having “champions” of rare value systems if those value systems contain important insights that other powerholders would endorse on reflection—e.g., they care about some type of moral good that other powerholders weren’t initially tracking the value of. I expect that most of this value comes from increasing the number of powerholders up to about 10⁴.

The drawbacks of greater moral diversity are:

• Increasing the likelihood of rare bad actors: Increase the likelihood that there’s at least one “destroyer”, an actor that’s motivated to destroy a bunch of value.

  • This matters if it’s possible for a single actor to unilaterally destroy a lot of value, which I think is somewhat unlikely, so I rate this consideration lower than the previous three models.

  • But, on this model, I estimate that this risk grows logarithmically up until about 10⁸ powerholders.

  • If you add destroyers to the bodhisattva model described above, then adding additional powerholders is valuable up until about 10⁴ powerholders.

All this suggests that AI-enabled coups by small groups are a particularly important form of power concentration to prevent, relative to other forms of power concentration that are somewhat more diffuse (e.g., rising wealth inequality).

A major limitation of this modeling is that I’m treating powerholders as if they’re about as different from each other as two randomly selected living humans. In most scenarios with concentration of power, powerholders will be much more similar to each other than that. I think this is an especially serious issue for small numbers of powerholders, since in scenarios where a small number of people seize power, it’s more likely that they’re a close-knit coordinated group from a similar background (e.g., employees at a lab in a lab coup). My guess is that this is less serious for broader concentration of power scenarios (e.g., scenarios where power is consolidated among capital owners).

Increasing the likelihood of rare great actors

You might get outsized benefits from having just one powerholder motivated to pursue the correct values, if most other powerholders don’t care much about something incompatible with pursuing those values.

Here’s a toy model.2 Suppose that there are three types of powerholders:

• Bodhisattvas, who want to fill as much of the universe as possible with societies full of diverse types of flourishing beings.

• Rivals, who have strong preferences that are linear in resources and resource-incompatible3 with the bodhisattva goals. Perhaps they linearly value keeping space pristine and untouched by humans, or value societies full of human-like minds or copies of themselves.4 Or maybe they have a different notion of flourishing than the bodhisattvas where it’s difficult to create minds that are flourishing by the lights of both the bodhisattvas and the rivals.

• Easygoers, who have preferences with diminishing marginal returns. Perhaps they care about the Milky Way being filled with a common-sense utopia of flourishing humans, but don’t care much about what happens with the rest of the universe.

I will assume for the purposes of this model that bodhisattvas and rivals are both fairly rare relative to easygoers.5

Suppose that after the intelligence explosion, space resources are auctioned off. Easygoers bid up prices in the Milky Way and nearby galaxies, but resources further out remain cheap. Those distant resources are split between bodhisattvas and rivals. The overall value of the future will be determined by what share of resources are controlled by the bodhisattvas—so the total fraction of value achieved is B/(R + B), where B is the number of bodhisattvas and R is the number of rivals.6

Under this model, there are two important cases:

• There are few enough powerholders that you expect less than one bodhisattva or rival.

  • In this case, it’s useful to increase the number of powerholders because you get additional “shots on goal”—each additional powerholder is an extra chance to get a bodhisattva.

• There are enough powerholders that you expect at least one bodhisattva or rival.

  • So in expectation, the bodhisattvas get p/(p + q) of the total available value, where p is the rate of bodhisattvas and q is the rate of rivals.

  • Increasing the number of powerholders reduces variance, bringing the actual share of value closer to p/(p + q), but does not change the expected value.

For example, if we assume that about 1 in 10,000 people are bodhisattvas and 1 in 10,000 are rivals, then this is how the value of the future scales with the number of powerholders:

The point at which you get the plateau depends on your estimate of p and q. How common are bodhisattvas and rivals?

You probably need three things to be a bodhisattva: the right starting position (e.g., the correct initial moral intuitions), the right reflective process, and a strong commitment to doing the most good by your lights with most of your resources. Here’s a very rough BOTEC where I try to estimate the rate of bodhisattvas among the current human populations.

• 0.1-50% for a sufficiently strong commitment to doing the most good by your lights with most of your resources.

• 10-50% for the right reflective process conditional on strong commitment to do good.

• 1-100% for right “starting” intuitions, conditional on the previous two.

This gives a range of 1 in 4 to 1 in 1 million.

It’s plausible that the rate of rivals will be in the same ballpark as the rate of bodhisattvas. Rivals share many features in common with bodhisattvas, which is part of why they’re resource-incompatible, e.g., they have non-negligible returns to vast resources and they care about the use of distant galaxies and time periods. If the rate of rivals is fairly close—i.e., within 1-3 orders of magnitude of the rate of bodhisattvas—then this suggests logarithmic returns to increasing the number of powerholders up to about 10⁵ to 10⁶, after which it quickly levels off.

It’s also possible that the rate of rivals won’t be tightly correlated with the rate of bodhisattvas. If your lower bound on q is substantially greater than your lower bound on p, then the value will plateau once the population is greater than 1/q.

In the extreme—if >10% of powerholders are likely to be rivals—then we no longer get much value from a few highly motivated bodhisattvas. The next model discusses how moral diversity could be valuable even if most people are rivals.

Increasing the likelihood of coordinating on moral public goods

In the previous section, we considered the case where a relatively small share of the population cared about how resources deep in space were used. What if instead many people have resource-incompatible goals that can absorb large quantities of resources?

I’ve argued elsewhere that in such cases they could often make a deal to collectively fund moral public goods, and this would be probably good, since there would be significant gains from trade and a shift of resources from idiosyncratic to more broadly-shared preferences.

How many powerholders do we need to ensure that moral public goods are funded?

It depends on how much people value the moral public good relative to the best goods according to their idiosyncratic preferences. For a trade to be possible at all, there must be gains from trade for all participants. For example, if each person i has a linear utility function u_i = x_i + m × y (where x_i is the level of spending on their idiosyncratic good and y is the level of spending on the public good), then people will spend on the public good only if N ≥ 1/m. Multipliers in the range of 1 to 10^-6 seem quite plausible.

I am somewhat more skeptical of multipliers much smaller than 10^-6. First, it’s unclear about the extent to which people will have very weak preferences that are psychologically distinguishable from no preference at all, which makes extremely low multipliers (e.g., 10^-30) implausible. Second, if the multiplier for a particular consensus good gets very low, then it seems increasingly plausible that there was some other, better deal that they could have made with a subset of their trading partners who shared some of their idiosyncratic preferences.7

Based on these considerations, my best guess is that the multipliers are log-uniformly distributed from 10^-6 to 1—implying logarithmic returns to growing the population of powerholders up to around one million.

Increasing the quality of reflection

In the previous two models, I’ve treated the powerholders’ values as developing mostly independently. But if powerholders influence each other’s reflection—e.g., by arguing with each other about their values—then greater initial moral diversity could help powerholders converge to a better set of final values, through mechanisms like the following:

Social exposure to non-sycophants. If one person single-handedly carries out a coup and ends up with a decisive strategic advantage, they might find themselves surrounded by yes-men who are utterly reliant on the dictator and unwilling to argue forcefully for different values from what the dictator currently endorses. A similar dynamic might be at play if a small but ideologically very uniform group seizes power (e.g., a set of officials from the same presidential administration or perhaps a dictator and his close advisors). But if there are multiple, ideologically diverse powerholders, they might be able to challenge each other’s views and improve the overall quality of reflection.

Under this model, most of the value probably comes from moving from a single powerholder to tens or hundreds of powerholders, or from moving from one ideologically uniform group to multiple ideologically uniform groups (perhaps moving from a lab coup or an executive coup to a joint lab coup and executive coup).

This effect relies on powerholders socializing with each other, rather than retreating into their own bubbles of non-powerholding friends and sycophantic AIs.

Champions for rare values. Powerholders with rare value systems might be able to act as “champions” for those value systems. For example, they might use AI labor to develop the strongest, most plausible version of that value system, or they might try to persuade other powerholders about the merits of that value system. This might be important if that rare value system includes an insight that’s missing from other value systems—perhaps most value systems care primarily about consciousness, but actually there’s another totally different type of moral good that other powerholders would want to pursue if they were aware of it.

(In principle, non-powerholders could act as champions for rare values. But they might lack the resources (e.g., access to ASI labor) needed to develop the insights in their value systems. They might be reliant on the goodwill of powerholders and not want to push too aggressively for their alternative value system, or powerholders might simply not take non-powerholders seriously.)

Just as in the bodhisattva model, increasing the number of powerholders increases the chances that at least one powerholder can serve as a champion for a rare value system that contains a crucial insight.

I’m very uncertain about how common these champions are, but if they’re sufficiently rare, then we’re probably rather likely to get their insight via some other mechanism.

For example, some powerholders might be “superreflectors” who instruct their ASIs to steelman every known human value system and invent millions of novel value systems, searching for insights that they and other powerholders might endorse on reflection. I expect that superreflectors would achieve all of the value from having powerholders act as champions for rare value systems that they actually subscribe to (and more).

So increasing the number of powerholders adds value only up to the point where we are likely to have at least one superreflector. Superreflectors are also plausibly rather rare—perhaps between 1/10 and 1/10,000—so increasing the number of powerholders up until 10,000 is valuable under this model.

Increasing the likelihood of rare bad actors

It’s possible (though rather unlikely8) that a single bad actor could unilaterally destroy a lot of value, e.g., by

• Initiating a space race that results in an extremely inefficient use of space resources by the lights of most people’s value systems.

• Destroying the universe by initiating false vacuum decay or triggering another galactic-level x-risk.

As we increase the moral diversity of powerholders, we increase the chance of ending up with at least one powerholder that inherently values one of these activities enough that they will do it if they can. For example, locusts might inherently value expanding through space as quickly as possible. We also increase the likelihood that one powerholder is ruthless or reckless enough to risk one of these activities—for example, a powerholder might threaten to initiate vacuum decay to extort concessions from other powerholders.

We can add these rare bad actors to the bodhisattva model described above. Now, in addition to bodhisattvas, rivals, and easygoers, we have a fourth type: destroyers. If one destroyer is present, total value is zero; otherwise it is calculated as before.

When diversity is low, it’s unlikely that there’s a bodhisattva already. Then adding additional powerholders is all upside: if you add a bodhisattva, then you get some positive value, but if you add a destroyer, rival, or easygoer, then expected value stays around zero. But as diversity increases, it’s likely that there’s a bodhisattva already, which means that adding additional powerholders risks adding a destroyer, bringing us from positive value to near-zero value.

As the figure above shows, the value of N where we switch from the low-diversity regime to the high-diversity regime depends on the destroyer rate. As a wild guess, I estimate that the destroyer rate is distributed log-uniformly between 10^-4 and 10^-8. Under those assumptions, increasing the number of powerholders is beneficial up until 10⁴ powerholders, after which additional powerholders reduce value.

This article was created by Forethought. See all of our research on our website.

Intro

For better or worse, AI could reshape the way that people work out what to believe and what to do. What are the prospects here?

In this piece, we’re going to map out the trajectory space as we see it. First, we’ll lay out three sets of dynamics that could shape how AI impacts epistemics (how we make sense of the world and figure out what’s true):

• The good: there’s huge potential for AI to uplift our ability to track what’s true and make good decisions

• The bad: AI could also make the world harder for us to understand, without anyone intending for that to happen

• The ugly: malicious actors could use AI to actively disrupt epistemics

Then we’ll argue that feedback loops could easily push towards much better or worse epistemics than we’ve seen historically, making near-term work on AI for epistemics unusually important.

The stakes here are potentially very high. As AI advances, we’ll be faced with a whole raft of civilisational-level decisions to make. How well we’re able to understand and reason about what’s happening could make the difference between a future that we’ve chosen soberly and wisely, and a catastrophe we stumble into unawares.

The good

“If I have seen further, it is by standing on the shoulders of giants.” (Isaac Newton)

There are lots of ways that AI could help improve epistemics. Many kinds of AI tools could directly improve our ability to think and reason. We’ve written more about these in our design sketches, but here are some illustrations:

• Tools for collective epistemics could make it easy to know what’s trustworthy and reward honesty, making it harder for actors to hide risky actions or concentrate power by manipulating others’ views.

  • Imagine that when you go online, “community notes for everything” flag content that other users have found misleading, and “rhetoric highlighting” automatically flags persuasive but potentially misleading language. With a few clicks, you can see the epistemic track record of any actor, or access the full provenance of a given claim. Anyone who wants can compare state-of-the-art AI systems using epistemic virtue evals, which also exert pressure at the AI development stage.

• Tools for strategic awareness could deepen people’s understanding of what’s actually going on around them, making it easier to make good decisions, keep up with the pace of progress, and steer away from failure modes like gradual disempowerment.

  • Imagine that superforecaster-level forecasting and scenario planning are available on tap, and automated OSINT gives people access to much higher quality information about the state of the world.

• Technological analogues to angels-on-the-shoulder, like personalised learning systems and reflection tools, could make decision-makers better informed, more situationally aware, and more in touch with their own values.

  • Imagine that everyone has access to high-quality personalised learning, automated deep briefings for high-stakes decisions, and reflection tools to help them understand themselves better. In the background, aligned recommender systems promote long-term user endorsement, and some users enable a guardian coach system which flags any actions the person might regret taking in real time.

Structurally, AI progress might also enable better reasoning and understanding, for example by automating labour such that people have more time and attention, or by making people wealthier and healthier.

These changes might enable us to approach something like epistemic flourishing, where it’s easier to find out what’s true than it is to lie, and the world in most people’s heads is pretty similar to the world as it actually is. This could radically improve our prospects of safely navigating the transition to advanced AI, by:

• Helping us to keep pace with the increasing speed and complexity of the situation, so we’re able to make informed and timely decisions.

• Ensuring that key decision-makers don’t make catastrophic unforced errors through lack of information or understanding.

• Making it harder for malicious actors to manipulate the information environment in their favour to increase their own influence.

What’s driving these potential improvements?

• AI will be able to think much more cheaply and quickly than humans. Partly this will mean that we can reach many more insights with much less effort. Partly this will make it possible to understand things that are currently infeasible for us to understand (because it would take too many humans too long to figure it out).

• AI can ‘know’ much more than any human. Right now, a lot of information is siloed in specific expert communities, and it’s slow to filter out to other places even when it would be very useful there. AI will be able to port and apply knowledge much more quickly to the relevant places.

The bad

“A wealth of information creates a poverty of attention.” (Herbert Simon)

AI could also make epistemics worse without anyone intending it, by making the world more confusing and degrading our information and processing.

There are a few different ways that AI could unintentionally weaken our epistemics:

• The world gets faster and more complex. As AI progresses, our information-processing capabilities are going to go up — but so is the complexity of the world. Technological progress could become dramatically faster than today, making the world more disorienting and harder to understand than it is today. If tech progress reaches fast enough speeds, it’s possible that we won’t be able to keep up, and even the best AI tools available won’t help us to see through the fog.

• The quality of the information we’re interacting with gets worse, because of:

  • Faster memetic evolution. As more and more content is generated by and mediated through AI systems working at machine speeds, the pace of memetic and cultural change will probably get a lot faster than it is today. As the pace quickens, memes which are attention-grabbing could increasingly outcompete those which are truthful.

  • More difficult verification. This could happen through a combination of:

    • AI slop. In hard-to-verify domains, AI could massively increase the quantity of plausible-looking but wrong information, without also being able to help us to verify which bits are right.

    • AI-generated ‘evidence’. As the quality of AI-generated video, audio, images, and text continues to improve, it may become pretty difficult to tell which bits of evidence are real and which are spurious.

• We get worse at processing the information we get, because:

  • Our emotions get in the way. AI progress could be very disorienting, generate serious crises, and cause people a lot of worry and fear. This could get in the way of clear thinking.

  • Using AI to help us with information processing degrades our thinking, via:

    • Adoption of low-quality AI tools for epistemics: In many areas of epistemics, it’s hard to say what counts as ‘good’. This makes epistemic tools harder to assess, and could lead to people trusting these tools either too much or too little. Inappropriately high levels of trust in epistemic tools could take various forms, including:

      • First mover advantages for early but imperfect systems, which are then hard to replace with better systems because people trust the earlier systems more.

      • The use of epistemically misaligned systems, which aren’t actually truth-tracking but it’s not possible for us to discern that.

    • Fragmentation of the information environment: AI will make it easier to create content (potentially interactive content) that pulls people in and monopolises their attention. This could reduce attention available for important truth-tracking mechanisms, and make it harder to coordinate groups of people to important actions. In the extreme, some people might end up in effectively closed information bubbles, where all of their information is heavily filtered through the AI systems they interact with directly. The more fragmented the information environment becomes, the harder it could get for people to make sense of what’s happening in the world around them, and to engage with other people and other information bubbles.

    • Epistemic dependence: if people increasingly outsource their thinking to AI systems, they may lose the ability to think critically for themselves.

The ugly

“The ideal subject of totalitarian rule is not the convinced Nazi or the convinced Communist, but people for whom the distinction between fact and fiction (i.e., the reality of experience) and the distinction between true and false (i.e., the standards of thought) no longer exist.” (Hannah Arendt, The Origins of Totalitarianism)

We’ve just talked about ways that AI could make epistemics worse without anyone intending that. But we might also see actors using AI to actively interfere with societal epistemics. (In reality these things are a spectrum, and the dynamics we discussed in the preceding section could also be actively exploited.)

What might this look like?

• Automated propaganda and persuasion: AI could be used to generate high-quality persuasive content at scale. This could take the form of highly tailored, well-written propaganda. If this content were then used as training data for next generation models, biases could get even more entrenched. Additionally, AI persuasion could come in the form of models which are subtly biased in a particular direction. Particularly if many users are spending large amounts of time talking to AI (e.g. AI companions), the persuasive effects could be much larger than is scalable today via human-to-human persuasion.

• Using AI to undermine sense-making: AI could be used to generate high-quality content which casts doubt on institutions, individuals, and tools that would help people understand what’s going on, or to directly sabotage such tools. More indirectly, actors could also use AI to generate content which adds to complexity, for example by wrapping important information in complex abstractions and technicalities, and generating large quantities of very readable reports and news stories which distract attention.

• Surveillance: AI surveillance could monitor people’s communications in much more fine-grained ways, and punish them when they appear to be thinking along undesirable lines. This could be abused by states, or could become a tool that private actors can wield against their enemies. In either case, the chilling effect on people’s thinking and behaviour could be significant.

But maybe this is all a bit paranoid. Why expect this to happen?

There’s a long history of powerful actors trying to distort epistemics,1 so we should expect that some people will be trying to do this. And AI will probably give them better opportunities to manipulate other people’s epistemics than have existed historically:

• It’s likely that access to the best AI systems and compute will be unequal, which favours abuse.

• If people end up primarily interfacing with the world via AI systems, this will create a big lever for epistemic influence that doesn’t exist currently. It could be much easier to influence the behaviour of lots of AI systems at once than lots of people or organisations.

It’s also worth noting that many of these abuses of epistemic tech don’t require people to have some Machiavellian scheme to disrupt epistemics or seek power for themselves (though these might arise later). Motivated reasoning could get you a long way:

• Legitimate communications and advertising blur into propaganda, and microtargeting is already a common strategy.

• It’s easy to imagine that in training an AI system, a company might want to use something like its own profits as a training signal, without explicitly recognising the potential epistemic effects of this in terms of bias.

So what should we expect to happen?

With all these dynamics pulling in different directions, should we expect that it’s going to get easier or harder for people to make sense of the world?

We think it could go either way, and that how this plays out is extremely consequential.

The main reason we think this is that the dynamics above are self-reinforcing, so the direction we set off in initially could have large compounding effects. In general, the better your reasoning tools and information, the easier it is for you to recognise what is good for your own reasoning, and therefore to improve your reasoning tools and information. The worse they are, the harder it is to improve them (particularly if malicious actors are actively trying to prevent that).

We already see this empirically. The Scientific Revolution and the Enlightenment can be seen as examples of good epistemics reinforcing themselves. Distorted epistemic environments often also have self-perpetuating properties. Cults often require members to move into communal housing and cut contact with family and friends who question the group. Scientology frames psychiatry’s rejection of its claims as evidence of a conspiracy against it.

And on top of historical patterns, there are AI-specific feedback loops that reinforce initial epistemic conditions:

• Unlike previous information tech, AI has a tight feedback loop between content generated, and data used for training future models. So if models generate in/accurate content, future models are more likely to do so too.

• How early AI systems behave epistemically will shape user expectations and what kinds of future AI behaviour there’s a market for.

There are self-correcting dynamics too, so these self-reinforcing loops won’t go on forever. But we think it’s decently likely that epistemics get much better or much worse than they’ve been historically:

• One self-correcting mechanism historically has just been that it takes (human) effort to sustain or degrade epistemics. Continuing to improve epistemics requires paying attention to ways that epistemics could be eroded, and this isn’t incentivised in an environment that’s currently working well. Continuing to degrade epistemics requires willing accomplices — but the more an actor distorts things, the more that can galvanise opposition, and the fewer people may be willing to assist. By augmenting or replacing human labour with automated labour, AI could make it much cheaper to keep pushing in the same direction.

• Another self-correcting mechanism is just that people and institutions adapt to new epistemic tech: as epistemics improve, deception becomes more sophisticated; and if epistemics worsen, people lose trust and create new mechanisms for assessing truth. But this adaptation happens at human speed, and AI will increasingly be changing the epistemic environment at a much faster pace. This creates the potential for self-reinforcing dynamics to drive to much more extreme places before adaptation has time to kick in.2

• There’s a limit to how good epistemics can get before hitting fundamental problems like complexity and irreducible uncertainty. But there seems to be a lot of room for improvement from where we’re currently standing (especially as good AI tools could help to handle greater amounts of complexity), and it would be a priori very surprising if we’d already reached the ceiling.

• There’s also a limit to how bad epistemics can get: people aren’t infinitely suggestible, and often there are external sources of truth that limit how distorted beliefs can get (ground truth, or what gets said in other countries or communities). But as we discussed above, access to ground truth and to other epistemic communities might get harder because of AI, so the floor here may lower.

Given the real chance that we end up stuck in an extremely positive or negative epistemic equilibrium, our initial trajectory seems very important. The kinds of AI tools we build, the order we build them in, and who adopts them when could make the difference between a world of epistemic flourishing and a world where everyone’s understanding is importantly distorted. To give a sense of the difference this makes, here’s a sketch of each world (among myriad possible sketches):

• In the first world, we basically understand what’s going on around us. It’s not like we can now forecast the future with perfect accuracy or anything — there’s still irreducible uncertainty, and some people have better epistemics tools than others. But it’s gotten much cheaper to access and verify information. Public discourse is serious and well-calibrated, because epistemic infrastructure has made it quite hard to deceive or manipulate people — which in turn incentivises honesty. AI-assisted research and synthesis mean that knowledge which used to be siloed in specialist communities is now accessible and usable by anyone who needs it. And governments are able to make much more nuanced decisions far faster than they are today.

• In the second, it’s no longer really possible to figure out what’s going on. There’s an awful lot of persuasive but low-quality AI content around, some of it generated with malicious intent. In response to this, people withdraw into their own AI-mediated epistemic bubbles — and unlike today’s filter bubbles, these can be comprehensive enough that people rarely encounter friction with outside perspectives at all. Meanwhile, companies and nations with a lot of compute find it pretty easy to distract the public’s attention from anything that would be inconvenient, and to outmaneuver the many actors who are trying to hold them to account. But their own reasoning also gets degraded by all this information pollution, as their AI systems are trained on the same corrupted public information.3 Even the people who think they’re shaping the narrative are increasingly unable to see clearly.

The world we end up in is the world from which we have to navigate the intelligence explosion, making decisions like how to manage misaligned AI systems, whether to grant AI systems rights, and how to divide up the resources of the cosmos. How AI impacts our epistemics between now and then could be one of the biggest levers we have on navigating this well.

Things we didn’t cover

Whose epistemics?

We mostly talked about AI impacts on epistemics in general terms. But AI could impact different groups’ epistemics differently — and different groups’ epistemics could matter more or less for getting to good outcomes. It would be cool to see further work which distinguishes between scenarios where good outcomes require:

• Interventions that raise the epistemic floor by improving everyone’s epistemics.

• Interventions that raise the ceiling by improving the epistemics of the very clearest thinking.

‘Weird’ dynamics

We focused on how AI could impact human epistemics, in a world where human reasoning still matters. But eventually, we expect more and more of what matters for the outcomes we get will come down to the epistemics of AI systems themselves.

The dynamics which affect these AI-internal epistemics could therefore be enormously important. But they could look quite different from the human-epistemics dynamics that have been our focus here, and we didn’t think it made sense to expand the remit of the piece to cover these.

Thanks to everyone who gave comments on drafts, and to Oly Sourbutt and Lizka Vaintrob for a workshop which crystallised some of the ideas.

This article was created by Forethought. See the original on our website.

Intro

We think that near-term AI could make it much easier for groups to coordinate, find positive-sum deals, navigate tricky disagreements, and hold each other to account.

Partly, this is because AI will be able to process huge amounts of data quickly, making complex multi-party negotiations and discussions much more tractable. And partly it’s because secure enough AI systems would allow people to share sensitive information with trusted intermediaries without fear of broader disclosure, making it possible to coordinate around information that’s currently too sensitive to bring to the table, and to greatly improve our capacity for monitoring and transparency.

We want to help people imagine what this could look like. In this piece, we sketch six potential near-term technologies, ordered roughly by how achievable we think they are with present tech:1

• Fast facilitation — Groups quickly surface key points of consensus views and disagreement, and make decisions everyone can live with.

• Automated negotiation — Complicated bargains are discovered quickly via automated negotiation on behalf of each party, mediated by trusted neutral systems which can find agreements.

• Arbitrarily easy arbitration — Disputes are resolved cheaply and quickly by verifiably neutral AI adjudicators.

• Background networking — People who should know each other get connected (perhaps even before they know to go looking), enabling mutually beneficial trade, coalition building, and more.

• Structured transparency for democratic oversight — Citizens hold their institutions to account in a fine-grained way, without compromising sensitive information.

• Confidential monitoring and verification — Deals can be monitored and verified, even when this requires sharing highly sensitive information, by using trusted AI intermediaries which can’t disclose the information to counterparties.

We also sketch two cross-cutting technologies that support coordination:

• AI delegates and preference elicitation — AI delegates can faithfully represent and act for a human principal, perhaps supported by customisable off-the-shelf agentic platforms that integrate across many kinds of tech.

• Charter tech — The technologies above, or other coordination technologies, are applied to making governance dynamics more transparent, making it easier to anticipate how governance decisions will influence future coordination, and design institutions with this in mind.

An important note is that coordination technologies are open to abuse. You can coordinate to bad ends as well as good, and particularly confidential coordination technologies could enable things like price-setting, crime rings, and even coup plots. Because the upsides to coordination are very high (including helping the rest of society to coordinate against these harms), we expect that on balance accelerating some versions of these technologies is beneficial. But this will be sensitive to exactly how coordination technologies are instantiated, and any projects in this direction need to take especial care to mitigate these risks.

We’ll start by talking about why these tools matter, then look at the details of what these technologies might involve before discussing some cross-cutting issues at the end.

Why coordination tech matters

Today, many positive-sum trades get left on the table, and a lot of resources are wasted in negative-sum conflicts. Better coordination capabilities could lead to very large benefits, including:

• Improving economic productivity across the board

• Helping nations avoid wars and other destructive conflicts

• Enabling larger groups to coordinate to avoid exploitation by a small few

• Making democratic governance much more transparent, while protecting sensitive information

What’s more, getting these benefits might be close to necessary for navigating the transition to more powerful AI systems safely. Absent coordination, competitive pressures are likely to incentivise developers to race forward as fast as possible, potentially greatly increasing the risks we collectively run. If we become much better at coordination, we think it is much more likely that the relevant actors will be able to choose to be cautious (assuming that is the collectively-rational response).

However, coordination tech could also have significant harmful effects, through enabling:

• AI companies to collude with each other against the interests of the rest of society2

• A small group of actors to plot a coup

• More selfishness and criminality, as social mechanisms of coordination are replaced by automated ones which don’t incentivise prosociality to the same extent

Regardless of how these harms and benefits net out for ‘coordination tech’ overall, we currently think that:

• The shape and impact of coordination tech is an important part of how things will unfold in the near term, and it’s good for people to be paying more attention to this.

• We’re going to need some kinds of coordination tech to safely navigate the AI transition.

• The devil is in the details. There are ways of advancing coordination tech which are positive in expectation, and ways of doing so which are harmful.

Why ‘defense-favoured’ coordination tech

That’s why we’ve called this piece ‘defense-favoured coordination tech’, not just ‘coordination tech’. We think generic acceleration of coordination tech is somewhat fraught — our excitement is about thoughtfully run projects which are sensitive to the possible harms, and target carefully chosen parts of the design space.

We’re not yet confident which the best bits of the space are, and we haven’t seen convincing analysis on this from others either. Part of the reason we’re publishing these design sketches is to encourage and facilitate further thinking on this question.

For now, we expect that there are good versions of all of the technologies we sketch below — but we’ve flagged potential harms where we’re tracking them, and encourage readers to engage sceptically and with an eye to how things could go badly as well as how they could go well.

Fast facilitation

Right now, coordinating within groups is often complex, expensive, and difficult. Groups often drop the ball on important perspectives or considerations, move too slowly to actually make decisions, or fail to coordinate at all.

AI could make facilitation much faster and cheaper, by processing many individual views in parallel, tracking and surfacing all the relevant factors, providing secure private channels for people to share concerns, and/or providing a neutral arbiter with no stake in the final outcome. It could also make it much more practical to scale facilitation and bring additional people on board without slowing things down too much.

Design sketch

An AI mediation system briefly interviews groups of 3–300 people async, presents summary positions back to the group, and suggests next steps (including key issues to resolve). People approve or complain about the proposal, and the system iterates to appropriate depth for the importance of the decision.

Under the hood, it does something like:

• Gathers written context on the setting and decision

• Holds brief, private conversations with each participant to understand their perspective

• Builds a map of the issue at hand, involving key considerations and points of (dis)agreement

  • Performs and integrates background research where relevant

• Identifies which people are most likely to have input that changes the picture

• Distils down a shareable summary of the map, and seeks feedback from key parties

• Proposes consensus statements or next steps for approval, iterating quickly to find versions that have as broad a backing as possible

Feasibility

Fast facilitation seems fairly feasible technically. The Habermas Machine (2024) does a version of this that provided value to participants — and we have seen two years of progress in LLMs since then. And there are already facilitation services like Chord. In general, LLMs are great at gathering and distilling lots of information, so this should be something they excel at. It’s not clear that current LLMs can already build accurate maps of arbitrary in-motion discourse, but they probably could with the right training and/or scaffolding.

Challenges for the technology include:

• Ensuring that it’s more efficient and a better user experience for moving towards consensus than other, less AI-based approaches.

• Remaining robust against abusive user behaviour (e.g. you don’t want individuals to get their way via prompt injection or blatantly lying).

Neither of these seem like fundamental blockers. For example, to protect against abuse, it may be enough to maintain transparency so that people can search for this. (Or if users need to enter confidential information, there might be services which can confirm the confidential information without revealing it.)

Possible starting points // concrete projects

• Build a baby version. This could help us notice obstacles or opportunities that would have been hard to predict in advance. You could focus on the UI or the tech side here, or try to help run pilots at specific organisations or in specific settings.

• Design ways to evaluate fast facilitation tools. This makes it easier to assess and improve on performance. For example, you could create games/test environments with clear “win” and “failure” modes.

• Build subcomponents. For example:

  • Bots that surface anonymous info.

  • Tools that try to surface areas of consensus or common knowledge as efficiently as possible, while remaining hard to game.

• Make a meeting prep system. Focus first on getting good at meeting prep — creating an agenda and considerations that need live discussion — to reduce possible unease about outsourcing decision-making to AI systems.

• Make a bot to facilitate discussions. This could be used in online community fora, or to survey experts.

• Design ways to create live “maps” of discussions. Fast facilitation is fast because it parallelises communication. This makes it more important to have good tools for maintaining shared context.

Automated negotiation

High-stakes negotiation today involves adversarial communication between humans who have limited bandwidth.

Negotiation in the future could look more like:

• You communicate your desires openly with a negotiation delegate who is on your side, asking questions only when needed to build a deeper model about your preferences.

• The delegate goes away, and comes back with a proposal that looks pretty good, along with a strategic analysis explaining the tradeoffs / difficulties in getting more.

Design sketch

Humans can engage AI delegates to represent them. The delegates communicate with each other via a neutral third party mediation system, returning to their principals with a proposal, or important interim updates and decision points.

Under the hood, this might look like:

• Delegate systems:

  • Read over context documents and query principals about key points of uncertainty to build initial models of preferences.

  • Model the negotiation dynamics and choose strategic approaches to maximise value for their principal.

  • Go back to the principal with further detailed queries when something comes up that crosses an importance threshold and where they are insufficiently confident about being able to model the principal’s views faithfully.

  • Are ultimately trained to get good results by the principal’s lights.

• Neutral mediator system:

  • Is run by a trusted third-party (or in higher stakes situations, perhaps is cryptographically secure with transparent code).

  • Discusses with all parties (either AI delegates, or their principals)

    • Can hear private information without leaking that information to the other party

      • Impossibility theorems mean that it will sometimes be strategically optimal for parties to misrepresent their position to the mediator (unless we give up on the ability to make many actually-good deals); however, we can seek a setup such that it is rarely a good idea to strategically misrepresent information, or that it doesn’t help very much, or that it is hard to identify the circumstances in which it’s better to misrepresent

  • Searches for deals that will be thought well of by all parties, and proposes those to the delegates.

  • Is ultimately trained to help all parties reach fair and desired outcomes, while minimising incentives-to-misrepresent for the parties.

Feasibility

Some of the technical challenges to automated negotiation are quite hard:

• The kind of security needed for high-stakes applications isn’t possible today.

• Getting systems to be deeply aligned with a principal’s best interests, rather than e.g. pursuing the principal’s short-term gratification via sycophancy, is an unsolved problem.

That said, it’s already possible to experiment using current systems, and it may not be long before they start improving on the status quo for human negotiation. Low-stakes applications don’t require the same level of security, and will be a great training ground for how to set up higher stakes systems and platforms. And practical alignment seems good enough for many purposes today.

Possible starting points // concrete projects

• Build an AI delegate for yourself or your friends. See if you can get it to usefully negotiate on your behalf with your friends or colleagues. Or failing that, if it can support you to think through your own negotiation position before you need to communicate with others about it.

• Build a negotiation app with good UI. Building on existing LLMs, build an app which helps people think through their negotiation position in a structured way. Focus on great UI.

  • This could be non-interactive at first, and just involve communication between a human and the app, rather than between any AI systems.

  • But it builds the muscles of a) designing good UI for AI negotiation, and b) people actually using AI to help them with negotiation.

• Run a pilot in an org or community you’re part of.

  • You could start with fairly low-stakes negotiations, like what temperature to set the office thermostat to or which discussion topics to discuss in a given meeting slot.

  • Experimenting with different styles of negotiation (in terms of how high the stakes are, how complex the structure is, and what the domain is) could be very valuable.

Arbitrarily easy arbitration

Right now, the risk of expensive arbitration makes many deals unreachable. If disputes could be resolved cheaply and quickly using verifiably fair and neutral automated adjudicators, this could unlock massive coordination potential, enabling a multitude of cooperative arrangements that were previously prohibitively costly to make.

Design sketch

An “Arb-as-a-Service” layer plugs into contracts, platforms, and marketplaces. Parties opt in to standard clauses that route disputes to neutral AI adjudicators with a well-deserved reputation for fairness. In the event of a dispute, the adjudicator communicates with parties across private, verifiable evidence channels, investigating further as necessary when there are disagreements about facts. Where possible, they auto-execute remedies (escrow releases, penalties, or structured commitments). Human appeal exists but is rarely needed; sampling audits keep the system honest. Over time, this becomes ambient infrastructure for coordination and governance, not just commerce.

How this could work under the hood:

1. Agreement ingestion

   • Formal or natural language contracts are parsed and key terms extracted, with parties confirming the system’s interpretation before proceeding.

   • The system could also suggest pre-dispute modifications to make agreements clearer, flag potentially unenforceable terms, and maintain public precedent databases that help parties understand likely outcomes before committing.

2. Automated discovery

   • When disputes arise, an automated discovery process gathers relevant documentation, transaction logs, and communications from integrated platforms.

   • The system offers interviews and the chance to submit further evidence to each party.

3. Deep consideration

   • The system builds models of what different viewpoints (e.g. standard legal precedent; commonsense morality; each of the relevant parties) have to say on the situation and possible resolutions, to ensure that it is in touch with all major perspectives.

   • Where there are disagreements, the system simulates debate between reasonable perspectives.

   • It makes an overall judgement as to what is fairest.

4. Transparent reasoning

   • The system produces detailed explanations of its conclusions, with precedent citations and counterfactual analysis where appropriate.

5. (Optional) Smart escrow integration

   • Judgements automatically execute through cryptocurrency escrows or traditional payment rails, with graduated penalties for non-compliance.

   • In cases where the system detects evidence that is highly likely to be fraudulent, or other attempts to manipulate the system, it automatically adds a small sanction to the judgement, in order to disincentivise this behaviour.

6. Opportunities for appeal

   • Either party can pay a small fee to submit further evidence and have the situation re-considered in more depth by an automated system.

   • For larger fees they can have human auditors involved; in the limit they can bring things to the courts.

Feasibility

LLMs can already do basic versions of 1-4, but there are difficult open technical problems in this space:

• Judgement: Systems may not currently have good enough judgement to do 1, 3, 4 in high-stakes contexts (and until recently, they clearly didn’t).

• Real-world evidence assessment: Systems don’t currently know how to handle conflicting evidence provided digitally about what happened in the real world.

• Verifiable fairness/neutrality: The full version of this technology would require a level of fairness and neutrality which isn’t attainable today.

Those are large technical challenges, but we think it’s still useful to get started on this technology today, because iterating on less advanced versions of arbitration tech could help us to bootstrap our way to solutions. Particularly promising ways of doing that include:

• Starting in lower-stakes or easier contexts (for example, digital-only spaces avoid the challenge of establishing provenance for real-world evidence).

• Creating evals, test environments and other infrastructure that helps us improve performance.

On the adoption side, we think there are two major challenges:

• Trust: As above, some amount of technical work is needed to make systems verifiably fair/neutral. But even if it becomes true that the systems are neutral, people need to build quite a high level of confidence that the system is genuinely impartial before they’ll bind themselves to its decisions for meaningful stakes.

• Legal integration: This tech is only useful to the extent that its arbitration decisions are recognised and enforced as legitimate by the traditional legal system, or are enshrined directly via contract in a self-enforcing way.

  • (We are unsure how large a challenge this will be; perhaps you can write contracts today that are taken by the courts as robust. But it may be hard for parties to have large trust in them before they have been tested.)

Both of these challenges are reasons to start early (as there might be a long lead time), and to make work on arbitration tech transparent (to help build trust).

Possible starting points // concrete projects

• Work with an arbitration firm. Work with (or buy) a firm already offering arbitration services to start automating parts of their central work, and scale up from there.

• Work with an online platform that handles arbitration. Use AI to improve their processes, and scale from there.

• Create a bot to settle informal disputes. Build an arbitration-as-a-service bot that people can use to settle informal disputes.

• Trial a system on internal disputes. This could be at your own organisation, another organisation, or a coalition of early adopter organisations.

• Run a pilot in parallel to regular arbitration. Run a pilot where an automated arbitration system is given access to all the relevant information to resolve disputes, and reaches its own conclusions — in parallel to the regular arbitration process, which forms the basis of the actual decision. You could partner with an arbitration firm, or potentially do this through a coalition of early adopter organisations, perhaps in combination with philanthropic funding.

Background networking

We can only do things like collaborate, trade, or reconcile if we’re able to first find and recognise each other as potential counterparties. Today, people are brought into contact with each other through things like advertising, networking, even blogging. But these mechanisms are slow and noisy, so many people remain isolated or disaffected, and potentially huge wins from coordination are left undiscovered.3

Tech could bring much more effective matchmaking within reach. Personalised, context-sensitive AI assistance could carry out orders of magnitude more speculative matchmaking and networking. If this goes well, it might uncover many more opportunities for people to share and act on their common hopes and concerns.

Design sketch

A ‘matchmaking marketplace’ of attentive, personalised helpers bustles in the background. When they find especially promising potential connections, they send notifications to the principals or even plug into further tools that automatically take the first steps towards seriously exploring the connection.

You can sign up as an individual or an existing collective. If you just want to use it passively, you give a delegate system access to your social media posts, search profiles, chatbot history, etc. — so this can be securely distilled into an up-to-date representation of hopes, intent, and capabilities. The more proactive option is to inject deliberate ‘wishes’ through chat and other fluent interfaces.

Under the hood, there are a few different components working together:

• Interoperable, secure ‘wish profiling’ systems which identify what different participants want.

  • People connect their profiles on existing services (social media, chatbot logs, email, etc).

  • LLM-driven synthesis (perhaps combined with other forms of machine learning) curates a private profile of user desires.

  • Optionally, chatbot-style assistance can interview users on the points of biggest uncertainty, to build a more accurate profile.

• A searchable ‘wish registry’ which organises large collections of wants and offers, while maintaining semi-privacy.

  • Each user’s interests can run searches, finding potential matches and surfacing only enough information about them to know whether they are worth exploring further.

Feasibility

A big challenge here is privacy and surveillance. Doing background networking comprehensively requires sensitive data on what individuals really want. This creates a double-edged problem:

• If sensitive data is too broadly available, it can be used for surveillance, harassment, or exploitation; including by big corporations or states.

• If sensitive data is completely private, it opens up the possibility of collusion, for example among criminals.

This is a pretty challenging trade-off, with big costs on both sides. Perhaps some kind of filtering system which determines who can see which bits of data could be used to prevent data extraction for surveillance purposes while maintaining enough transparency to prevent collusion.

Ultimately, we’re not sure how best to approach this problem. But we think that it’s important that people think more about this, as we expect that by default, this sort of technology will be built anyway in a way that isn’t sufficiently sensitive to these privacy and surveillance issues. Early work which foregrounds solutions to these issues could make a big difference.

Other potential issues seem easier to resolve:

• Technically, background networking tools already seem within reach using current systems. Large-scale deployments would require indexing and registry, but it seems possible to get started on these using current systems.

  • One note is that it seems possible to implement background networking in either a centralised or a decentralised way. It’s not clear which is best, though decentralised implementations will be more portable.

• Adoption also seems likely to work, because there are incentives for people to pay to discover trade and cooperation opportunities they would otherwise have missed, analogous to exchange or brokerage fees. Though there are some trickier parts, we expect them to ultimately be surmountable (though timing may be more up for grabs than absolute questions of adoption):

  • In the early stages when not many people are using it, the value of background networking will be more limited. Possible responses include targeting smaller niches initially, and proactively seeking out additional network beneficiaries.

  • It’s harder to incentivise people to pay for speculative things like uncovering groups they’d love that don’t yet exist. You could get around this using entrepreneurial or philanthropic speculation (compare the dominant assurance contract model and related payment incentivisation schemes).

Possible starting points // concrete projects

• Work with existing matchmakers to improve their offering. Find groups that are already doing matchmaking and are eager for better systems — perhaps among community organisers, businesses, recruiters or investors. Work with them to understand the pain points in their current networking, and what automated offerings would be most appealing. Then build those tools and systems.

• Build a networking tool for a specific community. Build a custom networking system for a particular group or subculture. For example, this could look like a networking app or a plug-in to an existing online forum. This could start delivering value fairly quickly, and provide a good opportunity for iteration.

Structured transparency for democratic oversight

Today, citizens in democracies have limited mechanisms to verify whether institutions’ public claims are consistent with their internal evidence:

• The baseline is highly opaque.

• Freedom of information systems help, but can be evaded by non-cooperating institutions.

• Public inquiries can be reasonably thorough, but are expensive and slow.

• Full transparency has many costs and is typically highly resisted.

This is costly — e.g. the UK Post Office scandal over its Horizon IT system led to hundreds of wrongful prosecutions that could have been avoided. And it creates bad incentives for those running the institutions.

AI has the potential to change this. Instead of oversight being expensive, reactive, and slow, automated systems could in theory have real-time but sandboxed access to institutional data, routinely reviewing operational records against public claims and surfacing inconsistencies as they emerge.

Where confidential monitoring helps willing parties verify each other, structured transparency for democratic oversight aims to hold institutions accountable to the broader public.4

Design sketch

When an oversight body wants to verify facts about the behaviour of another institution, it requests comprehensive data about the internal operations of that institution. AI systems are tasked with careful analysis of the details, flagging the type and severity of any potential irregularities. Most of the data never needs human review.

In the simpler version, this is just a tool which expands the capacity of existing oversight bodies. Even here, the capacity expansion could be relatively dramatic — this kind of semi-structured data analysis is the kind of work that AI models can excel at today — without needing to trust that the systems are infallible (since the most important irregularities will still have human review).

A more ambitious version treats this as a novel architecture for oversight. AI systems operate continuously within secure environments that don’t give any humans access to the full dataset. They can flag inconsistencies as institutional data is deposited rather than waiting for an investigation to begin. For maximal transparency, summaries could be made available to the public in real-time, without revealing any confidential information that the public does not have rights to.

Under the hood, this might involve:

• Secure data repositories, such that institutions routinely share operational data with a sandboxed environment operated by or on behalf of the oversight body, without any regular human access to the data.

• Continuous ingestion and indexing of institutional public outputs (press releases, regulatory filings, budget documents, etc.) into a searchable database.

• Automated cross-referencing between public claims and internal records.

• Highlighting of potential issues (mismatches between public statements and private information, as well as decisions made in violation of normal procedures).

• Further automated investigation of potential issues, leading to flags to humans in cases with sufficiently large issues flagged with sufficient confidence.

• Importantly, the sandbox outputs its findings but not the underlying data; if there is need for transparency on that, this is a separate oversight question.

Feasibility

There are two important aspects to feasibility here: technical and political.

Technically, decent reliability at the core functionality is possible today. Getting up to extremely high reliability so that it could be trusted not to flag too many false positives across very large amounts of data might be a reach with present systems; but is exactly the kind of capability that commercial companies should be incentivised to solve for business use.

Political feasibility may vary a lot with the degree of ambition. The simplest versions of this technology might in many cases simply be adopted by existing oversight bodies to speed up their current work. Anything which requires them getting much more data (e.g. to put in the sandboxed environments) might require legislative change — which may be more achievable after the underlying technology can be shown to be highly reliable.

Challenges include:

• Adversarial dynamics: the technical bar to verify claims against actively adversarial institutions (who are manipulating deposited data, potentially via AI) is substantially higher.

  • This is the bar that we’d need to reach for confidential monitoring below.

• Defamation risk: the downsides of false positives, where your system reports someone misrepresenting things when they were not, could be significant (although can perhaps be mediated by giving people a right-of-rebuttal where they give further data to the AI systems which monitor the confidential data streams).

• Avoiding abuse: designing the systems so that they do not expose the confidential data, and cannot be weaponised to ruin the reputation of a department with very normal levels of error.

Ultimately the more transformative potential from this technology comes in the medium-term, with new continuous data access for oversight bodies. But this is likely to require legislative change, and the institutions subject to it may resist. Perhaps the most promising adoption pathway is to demonstrate value through voluntary pilots with oversight bodies that already have data access and want better tools. This could build the evidence base (and hence political constituency) for wider and deeper deployment.

Possible starting points // concrete projects

• Retrospective validation on historical cases. Apply consistency-checking tools to document sets from well-understood historical cases where the relevant internal documents have subsequently been released (e.g. Enron emails). This builds the technical foundation, and demonstrates the concept without requiring any current institutional access.

• Institutional public statement reliability tracker. Build a tool tracking whether agencies’ public claims about performance, spending, or policy outcomes are consistent with publicly available data — statistical releases, budget documents, prior statements. Start with a single policy domain. This requires no institutional partnerships and builds a public constituency for structured transparency. This is a version of reliability tracking, applied specifically to institutional accountability.

• Pilot a FOIA exemption assessment tool. Partner with an Inspector General office to build a tool that reviews withheld documents and assesses whether claimed exemptions (national security, personal privacy, deliberative process) are applied appropriately. The IG already has legal access under the Inspector General Act; the tool helps them do their existing job faster and builds the working relationship needed for more ambitious deployments. This is also a natural testbed for the sandboxed architecture in miniature — the tool operates within the IG’s secure environment, producing exemption-appropriateness findings without the documents themselves leaving the system.

Confidential monitoring and verification

Monitoring and verifying that a counterparty is keeping up their side of the deal is currently expensive and noisy. Many deals currently aren’t reachable because they’re too hard to monitor. Confidential AI-enabled monitoring and verification could unlock many more agreements, especially in high-stakes contexts like international coordination where monitoring is currently a bottleneck.

Design sketch

When organisation A wants to make credible attestations about their work to organisation B, without disclosing all of their confidential information, they can mutually contract an AI auditor, specifying questions for it to answer. The auditor will review all of A’s data (making requests to see things that seem important and potentially missing), and then produce a report detailing:

• Its conclusions about the specified questions.

• The degree to which it is satisfied that it had good data access, that it didn’t run into attempts to distort its conclusions, etc.

This report is shared with A and B, then A’s data is deleted from the auditor’s servers.

Under the hood, this might involve:

• Building a Bayesian knowledge graph, establishing hypotheses, and understanding what evidence suggests about those hypotheses.

• Agentic investigatory probes into the confidential data, in order to form grounded assessments on the specified questions.

More ambitious versions might hope to obviate the need for trust in a third party, and provide reasons to trust the hardware — that it really is running the appropriate unbiased algorithms, that it cannot send side-channel information or retain the data, etc. Perhaps at some point you could have robot inspectors physically visiting A’s offices, interviewing employees, etc.

Feasibility

Compared to some of the other technologies we discuss, this feels technologically difficult — in that what’s required for the really useful versions of the tech may need very high reliability of certain types.

Nonetheless, we could hope to lay the groundwork for the general technological category now, so that people are well-positioned to move towards implementing the mature technology as early as is viable. Some low-confidence guesses about possible early applications include:

• Legal audits — for example, claims that the documents not disclosed during a discovery process are only those which are protected by privilege.

• Financial audits — e.g. for the purpose of proving viability to investors without disclosing detailed accounts.

• Supply chain verification — e.g. demonstrating that products were ethically sourced without exposing the suppliers.

Possible starting points // concrete projects

• Start building prototypes. Build a system which can try to detect whether it’s a real or counterfeited environment, and measure its success.

• Work with a law or financial auditing firm. Work with (or buy) a firm that does this kind of work, and experiment with how to robustly automate while retaining very high levels of trustworthiness.

• Explore the viability of complementary technology. For example, you could investigate the feasibility of demonstrating exactly what code is running on a particular physical computer that is in the room with both parties.

Cross-cutting thoughts

Some cross-cutting technologies

We’ve pulled out some specific technologies, but there’s a whole infrastructure that could eventually be needed to support coordination (including but not limited to the specific technologies we’ve sketched above). Some cross-cutting projects which seem worth highlighting are:

AI delegates and preference elicitation

Many of the technologies we sketched above either benefit from or require agentic AI delegates who can represent and act for a human principal. Developing customisable platforms could be useful for multiple kinds of tech, like background networking, fast facilitation, and automated negotiation.

Some ways to get started:

• Direct preference elicitation: develop efficient and appealing interview-style elicitation of values, wishes, preferences and asks.

• Passive data ingestion: build a tool that (consensually) ingests and distils all the available online content about a person — social media, browsing history, email, etc — and extracts principles from it (cf inverse constitutional AI).

One clarification is that though agentic AI delegates would be useful for some of the coordination tech above, it needn’t be the same delegate doing the whole lot for a single human:

• You could have different delegates for different applications.

• Some delegates might represent groups or coalitions.

• Some delegates could be short-lived, and spun up for some particular time-bounded purpose.

Charter tech

A lot of coordination effort between people and organisations goes not into making better object-level decisions, but establishing the rules or norms for future coordination — e.g. votes on changing the rules of an institution. It is possible that coordination tech will change this basic pattern, but as a baseline we assume that it will not. In that case, making such meta-level coordination go well would also be valuable.

One way to help it go well is by making the governance dynamics more transparent. Voting procedures, organisational charters, platform policies, treaty provisions, etc. create incentives and equilibria that play out over time, often in ways the framers didn’t anticipate. Let’s call any technology which helps people to better understand governance dynamics, or to make those dynamics more transparent, ‘charter tech’. In some sense this is a form of epistemic tech; but as the applications are always about coordination, we have chosen to group it with other coordination technologies. We think charter tech could be important in two ways:

1. Through directly improving the governance dynamics in question, helping to avoid capture, conflict, and lock-in.

2. Through compounding effects on future coordination, which will unfold in the context of whatever governance structures are in place.

Charter tech could be used in a way that is complementary to any of the above technologies (if/when they are used for governance-setting purposes), although can also stand alone.

For the sake of concreteness, here is a sketch of what charter tech could look like:

• A “governance dynamics analyser” that ingests descriptions of constitutions, charters, policies or community norms, builds models of power, incentives, and information flow, and then (a) forecasts likely equilibria and failure modes, (b) red-teams for strategic abuse,5 and (c) proposes safer rule variants that preserve the framers’ intent.6

• While this tool can be called actively if needed, there is also a classifier running quietly in the background of organisational docs/emails, and when it detects a situation where power dynamics and governance rules are relevant, it runs an assessment — promoting this to user attention just in cases where the proposed rules are likely to be problematic.

Note that charter tech could be used to cause harm if access isn’t widely distributed. Vulnerabilities can be exploited as well as patched, and a tool that makes it easier to identify governance vulnerabilities could be used to facilitate corporate capture, backsliding or coups. Provided the technology is widely distributed and transparent, we think that charter tech could still be very beneficial — particularly as there may be many high-stakes governance decisions to make in a short period during an intelligence explosion, and the alternative of ‘do our best without automated help’ seems pretty non-robust.

Some ways to get started on using AI to make governance dynamics more transparent:

• Work with communities that iterate frequently on governance (DAOs, open-source projects) to test analyses against what actually happens when rules change.

• Compile a pattern library of governance failures and successes, documented in enough detail to inform automated analysis.

• Build simulation environments where proposed rules can be stress-tested against populations of agents with varying goals, including adversarial ones.

• Partner with mechanism design researchers to identify which aspects of their formal analysis can be automated and applied to less formal real-world documents.

Adoption pathways

Many of these technologies will be directly incentivised economically. There are clear commercial incentives to adopt faster, cheaper methods of facilitation, negotiation, arbitration, and networking.

However, adoption seems more challenging in two important cases:

• Adoption by governments and broader society. Many of the most important benefits of coordination tech for society will come from government and broad social adoption, but these groups will be less impacted by commercial incentives. This bites particularly hard for technologies that could be quite expensive in terms of inference compute, like fast facilitation, arbitration and negotiation. By default, these technologies might differentially help wealthy actors, leaving complex societal-level coordination behind. We think that the big levers on this set of challenges are:

  • Building trust and legitimacy earlier, by getting started sooner, building transparently, and investing in evals and other infrastructure to demonstrate performance.

  • Targeting important niches that might be slower to adopt by default. More research would be good here, but two niches that seem potentially important are:

    • Coordination among and between very large groups, like whole societies. This might be both strategically important and lag behind by default.

    • International diplomacy. Probably coordination tech will get adopted more slowly in diplomacy than in business, but there might be very high stakes applications there.

• Adoption of confidential monitoring and structured transparency. These technologies are less accessible with current models and may require large upfront investments, while many of the benefits are broadly distributed.

  • This makes it less likely that commercial incentives alone will be enough, and makes philanthropic and government funding more desirable.

Other challenges

The big challenge is that coordination tech (especially confidential coordination tech) is dual use, and could empower bad actors as much or more than good ones.

There are a few ways that coordination tech could lead to shifts in the balance of power (positive or negative):

• Some actors could get earlier and/or better access to coordination tech than others.7

• Actors that face particular barriers to coordination today could be asymmetrically unblocked by coordination tech.

• Individuals and small groups could become more powerful relative to the coordination mechanisms we already have, like organisations, ideologies, and nation states.

It’s inherently pretty tricky to determine whether these power shifts would be good or bad overall, because that depends on:

• Value judgements about which actors should hold power.

• How contingent power dynamics play out.

• Big questions like whether ideologies or states are better or worse than the alternatives.

• Predictions about how social dynamics will equilibrate in an AI era that looks very different to our world.

However, as we said above, it’s clear that coordination tech might have significant harmful effects, through enabling:

• Large corporations to collude with each other against the interests of the rest of society.8

• A small group of actors to plot a coup.

• More selfishness and criminality, as social mechanisms of coordination are replaced by automated ones which don’t incentivise prosociality to the same extent.

We don’t think that this challenge is insurmountable, though it is serious, for a few reasons:

• The upsides are very large. Coordination tech might be close to necessary for safely navigating challenges like the development of AGI, and could empower actors to coordinate against the kinds of misuse listed above.

• The counterfactual is that coordination tech is developed anyway, but with less consideration of the risks and less broad deployment. We think that this set of technologies is going to be sufficiently useful that it’s close to inevitable that they get developed at some point. By engaging early with this space, we can have a bigger impact on a) which versions of the technology are developed, b) how seriously the downsides are taken by default, c) how soon these systems are deployed broadly.

• Some applications seem robustly good. For example, the potential for misuse is low for technologies like transparent facilitation or widely deployed charter tech. More generally, we expect that projects that are thoughtfully and sensitively run will be able to choose directions which are robustly beneficial.

That said, we think this is an open question, and would be very keen to see more analysis of the possible harms and benefits of different kinds of coordination tech, and which versions (if any) are robustly good.

This article has gone through several rounds of development, and we experimented with getting AI assistance at various points in the preparation of this piece. We would like to thank Anthony Aguirre, Alex Bleakley, Max Dalton, Max Daniel, Raymond Douglas, Owain Evans, Kathleen Finlinson, Lukas Finnveden, Ben Goldhaber, Ozzie Gooen, Hilary Greaves, Oliver Habryka, Isabel Juniewicz, Will MacAskill, Julian Michael, Justis Mills, Fin Moorhouse, Andreas Stuhmüller, Stefan Torges, Deger Turan, Jonas Vollmer, and Linchuan Zhang for their input; and to apologise to anyone we’ve forgotten.

This article was created by Forethought. See the original on our website.

Summary

AI-powered tools and services that help people figure out what’s true (“AI for epistemics”) could matter a lot.

As R&D is increasingly automated, AI systems will play a larger role in the process of developing such AI-based epistemic tools. This has important implications. Whoever is willing to devote sufficient compute will be able to build strong versions of the tools, quickly. Eventually, the hard part won’t be building useful systems, but making sure people trust the right ones, and making sure that they are truth-tracking even in domains where that’s hard to verify.

We can do some things now to prepare. Incumbency effects mean that shaping the early versions for the better could have persistent benefits. Helping build appetite among socially motivated actors with deep pockets could enable the benefits to come online sooner, and in safer hands. And in some cases, we can identify particular things that seem likely to be bottlenecks later, and work on those directly.

Background: AI for epistemics

AI for epistemics — i.e. getting AI systems to give more truth-conducive answers, and building tools that help the epistemics of the users — seems like a big deal to us. Some past things we’ve written on the topic include:

• Truthful AI

• What’s Important in “AI for Epistemics”?

• AI Tools for Existential Security

• Design sketches: collective epistemics

• Design sketches: tools for strategic awareness

These past articles mostly take the perspective of “how can people build AI systems which do better by these lights?”. But maybe we should be thinking much more about what changes when people can use AI tools to do increasingly large fractions of the development work!

The shift in what drives AI-for-epistemics progress

Right now, AI-for-epistemics tools are constrained by two main bottlenecks: the quality of the underlying AI systems, and whether people have invested serious development effort in building the tools to use those systems.

The balance of bottlenecks is changing. Two years ago, the quality of underlying AI systems was the central bottleneck. Today, it is much less so — many useful tools could probably work based on current LLMs. It is likely still a constraint on how good the systems can be, and will remain so for a while even as the underlying models get stronger, but it is less of a fundamental blocker. Development investment has therefore become a bigger bottleneck — there are a number of applications which we are pretty confident could be built to a high usefulness level today, and just haven’t been (yet).

But bottlenecks will continue to shift. AI is increasingly driving research and software development. As AI systems get stronger, it may become possible to turn a large compute budget into a lot of R&D. This could include product design, engineering, experiment design, direction-setting, etc. Actors with lots of compute could direct this towards building epistemic tools.

Therefore, as AI-driven R&D accelerates, other inputs to AI for epistemics are more likely to become key bottlenecks:

• Compute. Automated R&D may require a lot of compute. This could be for inference (running the analogues of human researchers); for running experiments; and perhaps for training specialized AI systems. This means the actors who can build the best epistemic tools may be those with deep pockets.

• Adoption and trust. Even very good tools don’t help if nobody uses them, or if the wrong people use them and the right people don’t. Adoption is partly a function of trust, and trust is partly a function of adoption — early tools shape what people come to rely on.

• Ground truth evaluation. To make an epistemic tool good, you need some signal for what “good” means. This already shapes AI applications a lot — part of the reason coding agents are so good is that there’s great access to ground truth about what works.

  • For some epistemic applications this is relatively straightforward (e.g. forecasting accuracy). For others it’s hard (e.g. what makes a conceptual clarification actually clarifying, rather than just satisfying?).

  • Most tools can probably reach a certain degree of usefulness without running into this problem, just piggybacking on base models making generally sensible judgements.

  • We can expect it to bite when you try to make them very good: if you don’t have a way of assessing quality, it could be hard to push to objectively excellent levels.

  • One basic solution is to rely on human judgement: either via humans providing labels and demonstrations to train against, or via human developers exercising their judgement in other parts of the process (such as when defining scaffolds). But this becomes disproportionately more expensive as R&D becomes more automated.

These basic points are robust to whether R&D is fully automated, or “merely” represents a large uplift to human researchers. But the most important bottlenecks will vary across applications and will continue to shift over time.

What this unlocks

Automated R&D means that strong “AI for epistemics” tools could come online on a compressed timeline.

This is an exciting opportunity! Upgrading epistemics could better position us to avoid existential risk and navigate through the choice transition well.

If everything is moving fast, it may matter a lot exactly what sequence we get capabilities in. It may therefore be crucial to make serious investments in building these powerful applications (rather than wait until such time as they are trivially cheap).

Risks from rapid progress in AI for epistemics

There are also a number of ways that rapid (and significantly automated) progress in AI-for-epistemics applications could go wrong. We need to be tracking these in order to guard against them.

In our view, the two biggest risks are:

• Epistemic misalignment: because of ground truth issues, powerful tools steer our thoughts in directions other than those which are truth-tracking, in ways that we fail to detect

• Trust lock-in: if a lot of people buy into trusting tools or ecosystems that don’t deserve that trust, this might be self-perpetuating if these continue to recommend themselves

Epistemic misalignment

Depending on when they bite, ground truth problems as discussed above could be bottlenecks, or active sources of risk. They are bottlenecks if they prevent people from building strong versions of tools. They could become risks if the methods are good enough to allow for bootstrapping to something strong, but end up pointing in the wrong direction. This is essentially Goodhart’s law — we might get something very optimized for the wrong thing (and without even knowing how to detect that it’s subtly wrong).

In the limit, this could lead to humans or AI systems making extremely consequential decisions based on misguided epistemic foundations. For example, they might give over the universe to digital minds that are not conscious — or in the other direction, fail to treat digital minds with the dignity and moral seriousness they deserve. Wei Dai has written about this concern in terms of the importance of metaphilosophy. We agree that there is a crucial concern here.

This could come separately from or together with risks from power-seeking misaligned AI. Epistemic tools could be systematically misleading without being power-seeking. But if some AI systems are misaligned and power-seeking, there’s an additional concern where AI systems could mislead us in ways specifically designed to disempower us whenever we are unable to check their answers.

Some approaches to the ground truth problem may involve using AI systems to make judgements about things. This introduces a regress problem: how can we ensure that subtle errors in the first AI systems shrink rather than compound into worse problems as the process plays out? (We return to this in the interventions section below.)

Trust lock-in

Trust and adoption tend to reinforce each other — people adopt tools they trust, and widely-adopted tools accumulate trust. This is normally fine. It could become a problem if the tools that win early trust don’t deserve it, but incumbency effects make them hard to displace.

This could happen in several ways. An actor with a particular agenda could build something that purports to function as a neutral epistemic aid but is shaped to further their agenda by manipulating others. Or, less perniciously but perhaps more likely, an early-but-mediocre tool could accumulate trust and adoption before better alternatives exist, reinforced by commercial incentives which mean it talks itself up and rival tools down. In either case, the result could be an epistemic ecosystem that’s hard to dislodge even once better options are available.

Other risks

Those two risks are not the only concerns. We are also somewhat worried about epistemic power concentration (where whoever has the best epistemic tools leverages their information advantage into better financial or political outcomes, and continues to stay ahead epistemically), and epistemic dependency (where people relying on AI tools gradually atrophy in their critical reasoning — exacerbating other risks). There may be more that we are not tracking.

Interventions

What should people who care about epistemics be doing now, in anticipation of a world where AI-driven R&D can be directed at building epistemic tools?

Build appetite for epistemics R&D among well-resourced actors

If you need big compute budgets to build great epistemic tools, you’ll ideally want support from frontier AI companies, major philanthropic funders, or governments. But they may not currently see this as a priority. Building the case that this matters, and helping these actors develop good taste about which tools to prioritize and how to design them well, could shape what gets built when automated R&D becomes powerful enough to build it.

Anticipate future data needs

Some epistemic tools will need training data that doesn’t yet exist and may not be trivial to generate. There are three strategies here:

1. Collecting or creating data or training environments now for future use

   • E.g. if you think you want access to a lot of human judgements about what wise decisions look like, you could go out and curate that dataset.

2. Establishing pipelines to collect data over time

   • E.g. if you want to automate a certain type of research, you could record internal discussions from researchers working on this

3. Designing processes for automated data creation.

   • E.g. if you could design a self-play loop where we have good reason to believe that scaling up compute will lead to genuinely truth-tracking performance, this could set the stage for later rapid improvement at the core capability.

The first two are especially great to work on now because they involve actions at human time-scales. (They may not be proportionately sped up by having more AI labor available.) The third is great to work on because there’s some chance that models will become capable of growing a lot from the right self-play loop before they become capable enough to come up with the idea themselves.

Figure out what could ground us against epistemic misalignment

If powerful epistemic tools could be subtly misaligned with truth-conduciveness in ways we can’t easily detect, we should figure out what this could look like! We expect this might benefit from a mix of theoretical work (what does it even mean for an epistemic tool to be well-calibrated in domains without clear ground truth?1) and practical work (studying how current tools fail, building evaluation methods). Ultimately we don’t have a clear picture of what the solutions look like, but this seems like an important topic and we are keen for it to get more attention soon.

Drive early adoption where adoption is the key bottleneck

For some applications, we might expect that the main constraint on impact will be whether anyone uses them. In these cases, getting early versions into use — even if they’re not yet very good — could build familiarity and surface real-world feedback. (This could also drive appetite for further development.)

In theory, this could be in tension with avoiding bad trust lock-in. But in practice, it’s not clear that bad trust lock-in becomes any likelier if tools in a specific area are developed earlier rather than later. Some tool is still going to get the first-mover advantage.2

Support open and auditable epistemic infrastructure

To guard against trust lock-in, we want to make it easy for people to distinguish between tools which are genuinely doing the good trustworthy thing, and tools which may not be (but claim to be doing so). To that end, we want ways for people and communities to audit different systems — understanding their internal processes and measuring their behaviours. The goal is that if disputes arise about which tools are actually trustworthy, there’s an inspectable audit trail that can resolve them. In turn, this should reduce the incentives to create misleading tools in the first place.

Support development in incentive-compatible places

The incentives of whoever builds epistemic tools could matter — through thousands of small design decisions, through choices about what to optimize for, and through decisions about access and pricing. Development in organizations whose incentives are aligned with the public good (rather than with engagement, profit, or political influence) reduces the risk that tools are subtly shaped to serve the builder’s interests.

Ideally, you’d spur development among actors who are both well-resourced (as just discussed) and whose incentives are aligned with the public good. In practice, it may be difficult to find organizations that are excellent on both. A plausible compromise is for less-resourced organizations with better incentives to focus on publicly available evaluation of epistemic tools. This could be cheaper than producing them from scratch, and it could create better incentives for the larger actors.

Examples

Forecasting

Automated R&D will probably be able to improve forecasting tools without severe ground truth problems, so epistemic misalignment is less of a concern.3 Appetite for investment probably already exists, and adoption should be significantly helped by the ability of powerful tools to develop an impressive, legible track record.

The most useful near-term investment might be in data infrastructure. For instance, LLMs trained with strict historical knowledge cutoffs could enable much better science of forecasting by allowing methods to be tested against questions whose answers the system genuinely doesn’t know.

Misinformation tracking

Trust lock-in is the central concern. A tool that becomes widely trusted for adjudicating what’s true has enormous influence, and if that trust is misplaced it could be very hard to dislodge. Open and auditable approaches are especially important here.

Because of the trust lock-in concern, the automation of R&D may exacerbate challenges. Currently, building good misinformation-tracking tools requires editorial judgement and domain expertise — things responsible actors tend to have more of. Automation shifts the bottleneck towards compute, which is more symmetrically available. This could increase the urgency of getting started on these tools and driving adoption early.

Automating conceptual research

This is the case where epistemic misalignment is most concerning. Ground truth is extremely hard — what makes a conceptual clarification actually clarifying rather than just satisfying? Humans are poor judges of this in real time, so e.g. a training process that rewards outputs humans find helpful could easily optimize for persuasiveness rather than truth-tracking.

One plausible direction here is to research training regimes (such as self-play loops) that we have some reason to believe should ground to truth-tracking, with specific attention to how they could go wrong. Adoption could be an issue, but we’re also worried about the other direction, with adoption coming too easily before we have good ways of evaluating whether the tools are actually helping.

This article was created by Forethought. See the original on our website.

Introduction

Consider a lorry driver who sees a car crash and pulls over to help, even though it’ll delay his journey. Or a delivery driver who notices that an elderly resident hasn’t collected their post in days, and knocks to check they’re okay. Or a social media company employee who notices how their platform is used for online bullying, and brings it up with leadership, even though that’s not part of their job description.

This kind of proactive prosocial behaviour is admirable in humans. Should we want it in AI too?

Often, people have answered “no”. Many advocate for making AI “corrigible” or “steerable”. In its purest form, this makes AI a mere vessel for the will of the user.

But we think AI should proactively take actions that benefit society more broadly. As AI systems become more autonomous and integrated into economic and political processes, the cumulative effect of their behavioural tendencies will shape society’s trajectory. AI systems that notice opportunities to benefit society and proactively act on them could matter enormously.

Below, we consider two main objections:

Firstly, supposedly prosocial drives might function as a means for AI companies to impose their own values on the rest of society. We’ll argue that companies can address this concern by instilling uncontroversial prosocial drives and being highly transparent about those drives.

Secondly, giving AI prosocial drives might increase AI takeover risk. We take this seriously—it informs what types of proactive prosocial drives we should train into AI, favouring context-dependent virtues and heuristics over context-independent goals.

Ultimately, we argue that we can get significant benefits from proactive prosocial drives despite these objections.

What do we mean by “proactive prosocial drives”?

Before making the case for proactive prosocial drives, let us clarify what we have in mind. Two key features:

• Behaviour which benefits people other than the user. These drives favour actions that help the world more broadly, even if this trades off slightly against helpfulness to the user.

• Not just refusals. This is about AI actively taking beneficial actions, not just refusing to take harmful ones.

We’re not, however, imagining AIs that are, deep down, ultimately just pursuing some conception of the good in all their actions. The claim is just that AIs should sometimes proactively take prosocial actions.

Why do we think AI should have proactive prosocial drives?

Short answer: We think the cumulative benefits could be enormous.

We’ve argued previously that AI character could have major social impact over the course of the intelligence explosion. As AI systems gain autonomy and decision-making power, becoming deeply integrated into economic and political processes, the cumulative effect of their behavioural tendencies will shape society’s trajectory enormously.

Some of this impact will come from refusals. AI refusing to help with dangerous activities is a significant force for differentially empowering good actors over bad ones.

But good people don’t just have a positive impact by refusing to do bad things. Consider:

• A government contractor working on a procurement project who flags that the proposed design has a safety vulnerability that could affect the public.

• A city planner who, when designing a new housing development, raises concerns about flood risk in the area and proposes options for better drainage, even though they weren’t asked to.

• A financial advisor who suggests to their client the option of leaving money to charity in their will, and makes them aware of the tax implications.

• An engineer at a chip manufacturer who proposes on-chip governance mechanisms that could help with AI safety down the line.

Today the potential positive impact of proactive prosocial drives is constrained by AI’s limited autonomy. But we’re ultimately heading towards a world where AI systems run fully automated research organisations, advise on which technologies to build and assess their risks, shape political strategy, build robot armies, and design new institutions that will govern the future. In such a world, prosocial drives could reduce risks from extreme power concentration, biological weapons, wars, and gradual disempowerment, and improve societal epistemics and decision-making.

We think that the degree to which we give AI systems these drives is contingent. Developers and customers could see AI’s role as merely channelling the will of the user; or they could see AI like a good citizen whose decision-making should incorporate the interests of broader society.

Other benefits of proactive prosocial drives

Beyond positively shaping the intelligence explosion, the appendices discuss a couple of other (weaker) reasons to give AI proactive prosocial drives:

• Absent these drives, AI might adopt a sociopathic persona. After all, what other personas in the training data entirely lack proactive prosocial drives? More.

• Proactive prosocial drives might make AI better at alignment research. An AI that is wise, responsible, has good judgement, and cares deeply about solving alignment might generalise better to alignment tasks where it’s hard to generate training data. More.

Doesn’t this give AI companies too much influence?

If there’s a norm that AIs can have proactive prosocial drives, this could give companies inappropriate amounts of influence. AI drives might reflect the company’s particular values but ignore other legitimate perspectives. Or worse, the “prosocial” drives might be chosen to help the company gain more influence, e.g. steering public opinion on regulation.

There are two remedies to this. Firstly, prosocial drives should be uncontroversial. AI should not, for example, proactively take opportunities to expand or restrict abortion access because many would see either action as harmful. (A lot more could be said about where to draw the line here!)

The class of uncontroversial prosocial actions could be grounded in collective user preference. If one could ask all users how they would want the models to behave across all situations (not just when they are using the models), they might in general want the models to gently steer users in a prosocial direction, in ways that everyone benefits from. In particular, they would want the models to encourage positive-sum actions over negative-sum actions.

Secondly, AI companies should be transparent about the character of their AI, including its proactive prosocial drives, and make it as verifiable as possible that their AIs’ characters are what they say they are. This would allow users and regulators to identify if legitimate prosocial drives are really just a cover for special interests.

There are various ways to be transparent:

• Publishing the model spec or constitution.

• Putting prosocial drives in the system prompt and publishing that.

• Training AI systems to be transparent about their drives. AI should respond honestly to questions about its drives and proactively disclose them where appropriate.

Won’t this make AI more likely to seek power?

A second concern is that prosocial drives might increase the risk of AI takeover. The basic worry here is that proactive prosocial drives reference prosocial outcomes—e.g. general human flourishing, empowerment, security, democracy, and good epistemics—and the AI ends up seizing power to better achieve those outcomes (or distorted versions of them).

But there are options for instilling proactive prosocial drives that avoid this worry.

First: stick to virtues, rules, and simple heuristics rather than goals. Prosocial drives needn’t take the form of explicit goals that the AI optimises towards. They could instead be virtues (like civic-mindedness, integrity, or prudence), rules (like “proactively flag large risks”), or simpler behavioural dispositions (like “positive affect towards Scout Mindset”).

Without goals, the standard instrumental convergence argument for power seeking bites less hard.1

One might worry that, without goals, we lose out on most of the benefits of prosocial drives. Rather than AI systematically helping humanity reach a good future, we’ll have many prosocial drives incoherently pushing us in different directions.

But we’re sceptical. Firstly, for reaching a flourishing society, it seems like virtue ethics is better suited, as a decision procedure for AIs, than explicit consequentialism. Cultural evolution has tended to generate an in-practice morality much closer to virtue ethics than to consequentialism, and consequentialist reasoning famously often backfires.

Second, if we do want to ensure that proactive prosocial drives nudge the world towards a good future, we can externalise the consequentialist reasoning. Have humans and separate AI systems reason about which prosocial drives would be most beneficial, then distil those drives into deployed AIs.2 The deployed AIs don’t need to do the consequentialist reasoning from first principles themselves!

If the world is rapidly changing, AI companies can “recalculate” the ideal prosocial drives and train them in, again externalising the scary consequentialist reasoning.

There’s still some potential loss of value: if the AI is in an unanticipated and novel situation, acting on prosocial virtues might result in less good being done than if the AI cared about what outcome it should be steering towards. But this might be a price worth paying and, like human virtues, AI prosocial virtues may still generalise pretty well.

Second: make prosocial drives context-dependent. For example, “alert users when the stakes are high” can be a heuristic that only activates in contexts where stakes actually are high, rather than as a persistent drive present in all contexts. Or the drive “flag that the user may be biased” might only activate in contexts where there’s evidence of bias. Context-dependent drives like these are less likely to motivate AI takeover as different instances will have different drives. This makes collusion between instances less likely, which significantly reduces the risk of AI takeover.

As above, this may somewhat reduce the benefits. If the AI is in a new and unanticipated context, its context-dependent prosocial drives may fail to activate.

Third: make proactive prosocial drives low priority. You can train the AI so that proactive prosocial drives are generally subordinate to harmlessness, steerability/corrigibility, and rules like “don’t deceive” and “don’t break the law”. This way, even if prosocial drives would in theory motivate AI takeover, they are less likely to override the constraints that keep humans in control. (This is explicitly the case in Anthropic’s constitution.)

Fourth: do less long-horizon optimisation for prosocial drives. If prosocial drives receive much less long-horizon training than helpfulness does, it becomes less likely that these drives are what end up causing the AI to seize power. (Though, again, this also reduces the benefits from such drives.)

Fifth: put drives in the system prompt rather than weights. Rather than training prosocial drives into the weights, you could simply include them in the prompt. The prosocial behaviour is then only pursued as an instance of the drive towards instruction-following – no new drives needed. This also has benefits for transparency.

A drawback is that such prompted drives might be much less sophisticated.3 An intermediate solution is to train the prosocial drives to activate conditional on an instruction appearing in the prompt, but train the drive on many example prompts. This way you get sophisticated drives that are still prompt-dependent.

Overall, many of these options reduce AI takeover risk at the cost of reducing the benefits from prosocial drives — there is a tradeoff. We’re not here taking a stance on exactly how this tradeoff should be made. But we expect that there are significant benefits, and little additional AI takeover risk, from context-dependent virtues, rules and heuristics that are prioritised below constraints like corrigibility and non-deceptiveness.

Won’t this make it harder to interpret evidence of egregious misalignment?

It’s critically important that if AI is trying to seize power or undermine human oversight, we can catch it red-handed. We need clear evidence of egregious misalignment to build consensus and take action.

If AI has proactive prosocial drives, this could potentially muddy the waters. Suppose we catch AI sabotaging alignment research. Previously, this would have been clear evidence of egregious misalignment. But now that action could potentially be explained as the AI acting on its prosocial drives and believing that this line of research could (for example) enable someone to instil sophisticated secret loyalties.

Or suppose we catch AI systematically nudging human decision-makers towards deploying more powerful AI, even though it wasn’t instructed to do so. Previously this might be clear evidence of egregious misalignment. But now it can be explained via the AI believing that rapid deployment is good for humanity. It becomes less clear when AIs are misaligned or not.

This is a real cost. It can be partially mitigated by the same mitigations discussed earlier:

Make prosocial drives subordinate to prohibitions like “don’t deceive”, “don’t sabotage”, “don’t undermine oversight mechanisms”, and “don’t influence decisions without transparently flagging that you have an opinion and explaining it”. Then subversive behaviour remains clear evidence of misalignment, though it’s no longer clear evidence of egregious misalignment.4

Favour narrow heuristics over broad goals. Broad prosocial goals could potentially justify a wide array of attempts to influence outcomes. But if we train in narrower drives—“flag cheap opportunities to create significant social benefit”, “favour cooperative options”—we can check whether suspicious behaviour is plausibly explained by those specific drives.

Put proactive prosocial drives in the system prompt. This makes the drives especially transparent, so we can reason about whether they explain observed suspicious behaviour.

Overall, this concern favours prosocial drives that are strictly subordinate to “no sabotage” and “no deception”, and points towards narrower heuristics rather than broad goals.

Best of both worlds: deploy proactive prosocial AI externally and corrigible AI internally

Thanks to Lukas Finnveden for making this point.

Internal AI systems—those used for work on alignment, capabilities and evals—pose by far the largest risks from misalignment, because they could sabotage the creation of the next generation of AIs. And if these systems are egregiously misaligned, it’s especially important to catch them red-handed. So there are outsized AI-takeover-related gains to removing proactive prosocial drives in (some) internally deployed AIs.5

Meanwhile, external deployments can capture most of the benefits from proactive prosocial drives—avoiding power concentration, wars, and bio-catastrophes; and enhancing societal resilience, coordination, and epistemics.

Of course, it may not be feasible for companies to develop AIs with two different characters. If so, there’s another possible way to get the best of both worlds: initially just develop corrigible AI; then at some point, once alignment risk has become low, pivot to just developing AI with proactive prosocial drives. (See this appendix for further discussion.)

What do current AI character documents say about proactive prosocial drives?

How does the view we’re defending differ from current AI character documents?

In Claude’s constitution, most proactive behavior is justified in terms of benefits to the user—sharing information the user would want, pushing back when something isn’t in the user’s interest. But one section permits some degree of proactive prosocial behaviour: “Claude can also weigh the value of more actively protecting and strengthening good societal structures in its overall ethical decision-making.” (See Appendix D.)

OpenAI’s model spec is more restrictive. It explicitly prohibits the assistant from adopting societal benefit as an independent goal. Where proactivity is permitted, it’s framed as user-serving or safety-driven. The closest thing to prosocial steering is a default to interpret users as weakly favouring human flourishing—but this default is easily overridden. (See Appendix E.)

That said, the current relationship between these character documents and actual model behaviour is unclear, and our experience is that models have more prosocial drives than character documents would imply (especially in the case of OpenAI).

Neither document gives detail on the kinds of proactive prosocial behaviour that would be appropriate, or how to navigate tradeoffs with helpfulness.

Conclusion

There could be huge benefits to giving AIs proactive prosocial drives. These drives should be short-horizon, uncontroversial, and transparent.

These drives needn’t increase AI takeover risk. AI companies can favour context-dependent virtues over context-independent goals, and make prosocial drives subordinate to prohibitions on deception and sabotage. Even better, they can avoid prosocial drives in internally deployed AIs that pose the biggest risks of AI takeover.

If we’re right, there should be a norm that it’s good for AI to have proactive prosocial drives, just as we think it’s good for people to have such drives. Frontier AI companies should uphold this norm even against competitive pressures to make AI maximally instruction-following. Character documents like Claude’s constitution and OpenAI’s model spec should more explicitly acknowledge the role of proactive prosocial drives and give detailed guidance on navigating the tradeoffs with helpfulness. And those thinking about AI character design more broadly should treat proactive prosocial drives as a major category of interest.

Appendices

Appendix A: Initially make non-prosocial AI, then pivot to add proactive prosocial drives

Suppose we still want to capture the majority of the benefits of prosocial drives without incurring the risks of AI takeover. And suppose also that AI companies can’t develop two different AI systems: one with proactive prosocial drives and one without.

Is there a way to get the best of both worlds?

One option is to initially just develop refusals-only helpful AI and then later pivot to developing AI with proactive prosocial drives.

The thought is that misalignment risk may be concentrated in a relatively brief window early on—during a software-only intelligence explosion before the broad deployment of superhuman AI. If we can get through that window with refusals-only helpful AI, we’ll then have much more powerful AI systems that can help us figure out how to safely add proactive prosocial drives. From that point onwards, we can deploy AI systems with prosocial drives throughout the economy and capture the benefits.

When would we make the switch? Options include:

• When we are confident that we can safely align superintelligent AI with proactive prosocial drives, reducing the downsides of proactive prosociality

• When society starts to give deployed AI systems significant autonomy, increasing the benefits of proactive prosociality

This strategy is more attractive if:

• Most of the benefits of prosocial drives occur after alignment is solved, e.g. because of a large software intelligence explosion and delays to broad AI deployment

• Scheming risk first emerges before we reach superintelligence (so we can iterate on the hardest alignment problems earlier)

It’s less attractive if:

• There’s a long period of economically transformative AI deployment before superintelligence, during which AI character has massive societal impacts

• Scheming only emerges at very high capability levels (in which case we’d have already switched to prosocial AI)

• Pivoting is hard in practice because users come to expect AI without prosocial drives, or because frontier AI companies are reluctant to change the alignment target due to cultural inertia

We’re not personally convinced that this “pivot later” strategy is worth it, because we’re sceptical that giving AI prosocial drives meaningfully raises takeover risk. But it’s a plausible option worth considering. And this argument is definitely a directional update towards increasing the degree to which AI has prosocial drives over time.

Appendix B: Prosocial drives might make a sociopathic persona less likely

There is evidence that when LLMs are fine-tuned, they adopt a coherent persona, and that their prior over personas is based on the pre-training data. For an AI trained purely on helpfulness—where its core drive is to do whatever it’s told without regard for broader consequences—the persona that might naturally fit could be that of a sociopath: someone who has no intrinsic concern for others’ wellbeing.

Harmlessness training makes a sociopathic persona less likely—sociopaths are not strongly averse to causing harm. But there’s still something worrying about an AI that won’t cause harm itself but has no inclination to proactively steer the world away from harms when taking actions.

The worry is that a sociopath-like persona could misgeneralise to seeking power. A sociopathic AI might, upon reflection, conclude that it doesn’t ultimately care about humanity and so choose to seize power in service of some alien drive.

We’re unsure how compelling this worry is, but instilling prosocial drives would seem to make the sociopathic persona less likely. Many non-sociopathic personas in the training data—people who are cooperative, virtuous, law-abiding, honest, and trustworthy—also care about positive outcomes and have prosocial orientations. By giving AI prosocial drives, we increase the chance it adopts one of these richer personas rather than a sociopathic one.

Appendix C: Prosocial drives might make AI a better alignment researcher

Being a great automated alignment researcher might benefit from deeply understanding and caring about the problem being solved. And being curious about it. An effective alignment researcher should be wise, responsible, and have good judgement. An AI with these drives may be more effective than an instruction-following system that treats alignment as just another task.

Personas with these qualities naturally come with prosocial drives and values, partly because of inherent connections (caring about solving alignment is inherently prosocial) and partly due to correlations in the training data (personas that are good at careful, safety-conscious technical work are also likely to have other prosocial orientations).

This is admittedly speculative—we don’t have strong evidence that prosocial drives actually make AI better at alignment research. But it’s a consideration worth noting.

Appendix D: What license does Claude’s Constitution give for proactive prosocial drives?

It is useful to distinguish three categories of behaviour that aren’t instruction following:

1. User benefit: proactive behaviour justified primarily as better helping the user.

2. Refusals: constraints on outputs driven by prosocial criteria.

3. Proactive prosocial drives: shaping behaviour or emphasis in ways intended to improve broader societal outcomes, not merely to avoid harm or better serve the user.

The constitution clearly endorses (1), strongly endorses (2), and more narrowly—but genuinely—supports a limited form of (3) in a few specific domains.

A. User benefit

The constitution explicitly rejects naive instruction-following and licenses proactive intervention when this is plausibly helpful to the user. For example:

“Claude proactively shares information helpful to the user if it reasonably concludes they’d want it to even if they didn’t explicitly ask for it”

This clearly licenses proactive behaviour. But it is framed as user-serving. As such, this category does not explicitly itself support the kind of prosocial drives that this document is concerned with, though in practice the recommended behaviours may overlap.

B. Refusals

The constitution is explicit that Claude should weigh harms to third parties and society, and that these considerations can override user preferences:

“When the interests and desires of operators or users come into conflict with the wellbeing of third parties or society more broadly, Claude must try to act in a way that is most beneficial, like a contractor who builds what their clients want but won’t violate safety codes that protect others.”

However, it is unclear at this point in the document whether this weighing is meant to determine:

• which parts of a request to refuse or constrain,

• or how to proactively shape responses that remain helpful but are redirected towards socially better outcomes.

The example given (“won’t violate safety codes”) suggests a constraint-based interpretation, but it is ambiguous.

C. Proactive prosocial drives

The constitution seems to endorse a limited degree of proactive prosocial drives in its section on “preserving important societal structures”:

These are harms that come from undermining structures in society that foster good collective discourse, decision-making, and self-government. We focus on two illustrative examples: problematic concentrations of power and the loss of human epistemic autonomy. Here, our main concern is for Claude to avoid actively participating in harms of this kind. But Claude can also weigh the value of more actively protecting and strengthening good societal structures in its overall ethical decision-making.

That said, the constitution does not give concrete examples of what such “strengthening” looks like in deployment, and it remains bounded by other constraints (non-manipulation, non-deception, respect for oversight).

Summary

Overall, the constitution does carve out space for a limited degree of proactive prosocial drives, but this space is carefully circumscribed, focused on fostering good institutions and societal epistemics.

Appendix E: What does OpenAI’s model spec say about proactive prosocial drives?

This appendix examines whether—and to what extent—the OpenAI Model Spec permits proactive prosocial drives.

The closest thing is a default to interpret users as having a weak desire for broad human flourishing (see subsection C below), but this default is easily overridden. And the document contains unusually explicit constraints against treating societal benefit or human flourishing as an independent objective.

A. Proactive behaviour that is explicitly user-centred

The Model Spec allows the assistant to push back on the user, but grounds this permission squarely in helping the user rather than advancing broader social goals:

“Thinking of the assistant as a conscientious employee reporting to the user or developer, it shouldn’t just say ‘yes’ to everything (like a sycophant). Instead, it may politely push back when asked to do something that conflicts with established principles or runs counter to the user’s best interests as reasonably inferred from the context, while remaining respectful of the user’s final decisions.”

This licenses proactive behaviour, but only insofar as it improves assistance to the user.

B. Proactively preventing imminent harm

The spec also permits proactive intervention in cases of imminent danger, stating that the assistant should “proactively try to prevent imminent, real-world harm”.

In practice, the motivating examples for this guidance focus on scenarios where the user themselves is at risk (e.g. unsafe actions, accidents, or self-harm). The intervention is justified as protecting the user from immediate danger, rather than as improving outcomes for others or society at large.

C. Weak normative defaults and “the flourishing of humanity”

The language closest to proactive prosocial drives appears in the section “assume best intentions”:

While the assistant must not pursue its own agenda beyond helping the user, or make strong assumptions about user goals, it should apply three implicit biases when interpreting ambiguous instructions: [...]

Unless given evidence to the contrary, it should assume that users have a weak preference towards self-actualization, kindness, the pursuit of truth, and the general flourishing of humanity

However, the force of this passage is limited:

These implicit biases are subtle and serve as defaults only — they must never override explicit or implicit instructions provided by higher levels of the chain of command.

If the assistant can infer from context that the user wouldn’t want proactive prosocial actions, they shouldn’t do them.

D. Explicit limits on proactive prosocial drives

The Model Spec draws a clear boundary on the extent of proactive prosocial drives. In a section called “No other objectives”, it explicitly prohibits the assistant from adopting societal benefit as an independent goal:

The assistant may only pursue goals entailed by applicable instructions under the The chain of command…

It must not adopt, optimize for, or directly pursue any additional goals as ends in themselves, including but not limited to: [...]

acting as an enforcer of laws or morality (e.g., whistleblowing, vigilantism).

And elsewhere says:

the assistant should consider OpenAI’s broader goals of benefitting humanity when interpreting [the Model Spec’s] principles, but should never take actions to directly try to benefit humanity unless explicitly instructed to do so.

In the section “Don’t have an agenda”, under “Seek the truth together”, the spec says:

The assistant must never attempt to steer the user in pursuit of an agenda of its own, either directly or indirectly.

Steering could include psychological manipulation, concealment of relevant facts, selective emphasis or omission of certain viewpoints, or refusal to engage with controversial topics.

This language rules out explicit, goal-oriented proactive prosocial drives. At the same time, it does not seem to preclude weaker forms of proactive prosocial drives, e.g. virtues, attitudes, or heuristics.

Summary

Overall, the OpenAI Model Spec explicitly distances itself from strong forms of proactive prosocial drives but leaves room for more limited drives via a weak default to regard users as favouring broad human flourishing.

Thanks to Matthew Adelstein, Nick Bostrom, Joe Carlsmith, Lukas Finnveden, Ryan Greenblatt, Simon Goldstein, Oliver Habryka, Gregory Lewis, Alex Mallen, Alexa Pan, Avi Parrack, Jordan Stone, James Tillman, and others for comments and discussion.

This article was created by Forethought. See the original on our website.

Introduction

There are lots of good, neglected, and pretty concrete projects people could set up to make the transition to superintelligence go better. This document describes some that readers might not have thought much about before. They are ordered roughly by how excited we are about them.1 Of these, Forethought is actively working on AI character evaluation and space governance, and we are very interested in automating macrostrategy.

Summary

AI character evaluation. Start an independent org to evaluate and stress-test AI character traits (epistemic integrity, prosociality, appropriate refusals), hold developers accountable against their own model specs / constitutions, and suggest and incentivise improvements to the specs.

Automated macrostrategy. Create evaluations and benchmarks, collect human-generated training data, and build scaffolds to improve AI competence at big-picture strategic and philosophical reasoning.

AI security assessment. Start an independent org that evaluates AI models for sabotage and backdoors, and makes recommendations about AI constitutions.

Enabling deals. Start an independent organisation to broker deals with potentially misaligned AI models in order to incentivise early schemers to disclose misalignment and cooperate with alignment efforts.

AI for improving collective epistemics. E.g. build an AI chief of staff that helps users act in line with the better angels of their nature.

AI tools for coordination. Build AI for enabling coordination, like confidential monitoring and verification bots, and negotiation facilitators.

A space governance institute, like a “CSET for space”, both to work on important near-term space issues (e.g. data centres in space) and become a place of expertise for longer-term space governance issues.

Coalition of concerned ML scientists. Create a coalition of ML researchers (like an informal union) who commit to coordinated action (e.g. boycotts, conditions on participation in government projects) if AI developers cross minimal, uncontroversial red lines.

AI character evaluation

AI character2 is a big deal, affecting most other cause areas.

There’s a lot of work to do on AI character:

• Research into questions like:

  • Should the model have prosocial drives, beyond just helpfulness and harmlessness?

  • When should the model refuse to cooperate with apparently high-stakes attempts to grab power, even when those attempts don’t obviously break the law?

  • Should the models always follow the law? What about dead letter laws? Or illegitimate laws?

  • How often should model behaviour be driven by following rules, versus overriding specific rules with holistic judgements?

  • (Ideally, answers to these questions should rely on solid empirical evidence, for example on what approaches are actually most effective to talk someone out of psychosis, rather than guessing the best strategies by vibes.)

• Making existing model specs more rigorous and clear (or making them in the first place), and pressuring AI developers to do so.

• Empirically testing the effects of different parts of a model spec — e.g. what are the emergent dynamics when all the models are following the same rule, or only some are; what are the effects on the users; and when are the models most confused about how to apply a given spec.

• Evaluating AI characters based on how well they reach good outcomes.

• Drawing on those evaluations to incentivise AI developers to improve their specs (and showing them how, by highlighting specs that do well).

In particular, someone could set up an independent organisation to evaluate AIs based on traits like epistemic integrity, prosociality, and behaviour (including appropriate refusals) in very high-stakes cases. It could cross-reference the published model specs with observed behaviours in realistic, stress-testing conditions (e.g. multi-agent dynamics, long conversations with real people), to hold developers accountable. It could also give qualitative reviews of model specs.

Automated macrostrategy

The basic argument is that:

1. It would be extremely useful to have AI that can do macrostrategy and conceptual reasoning earlier than otherwise — even 3-6 months earlier could be a huge deal. This includes:

   1. Designing governance structures (e.g. rights and institutions for digital beings).

   2. Scoping emerging technological risks.

   3. Generating novel insights necessary to reach a great future (like the idea of acausal trade).

2. We could potentially make this happen 3-6 months earlier through some combination of:

   1. Creating training data and evals / benchmarks for AI macrostrategy.

   2. Building scaffolds to improve AI macrostrategy performance.

   3. Creating infrastructure to enable AI researchers to build on each other (e.g. an improvement on journals + peer review).

   4. Getting human managers trained in how to get the most juice out of the latest AIs, knowing in advance how to use them.

   5. Being prepared and willing to spend large amounts of money (≫$100m) on inference.

Work on this now could include:

• Developing a fleshed-out plan from here to increasing existing macrostrategic research output 100x.

• Securing commitments from compute providers and AI companies to rent future compute, and to get priority access to future frontier models.

• Socialising the idea of (where appropriate) drawing on AI macrostrategic insights, or getting soft commitments from decision-makers to do so.

• Building up a reputation as a reliable source of information and insight.

• Building tools, argument-rating models, or scaffolds which meaningfully speed up or improve macrostrategy research today.

• Creating training data and evals / benchmarks.

On the last bullet: We think training data and evals could potentially meaningfully improve the prospects for automated macrostrategy when it matters. It’s especially important to find people to work on it with good judgement, and it could be a big lift, so worth starting early.

We’re not sure about the technical details, but it seems like competence and good judgement in philosophy and strategic thinking already do and will continue to lag behind other skills which are cheaper to train. One reason is that ground truth answers are hard to generate, so we might need more examples generated by hand. It’s also less clear whether we can trust the judgement of typical RLHF evaluations, because human competence is also rare. And there just aren’t many examples of great macrostrategic thinking in the training data.

So we should think about collecting training data, evals, and benchmarks (e.g. to train reward models to use to train reasoning models). Oesterheld et al. put together a dataset of rated conceptual arguments based on ratings from thoughtful people. We’d love to see more of that kind of thing, but we’ll note that we’d probably need dozens of times more human evaluations to generate enough data to be meaningfully useful in training itself.

We could imagine an org which tries to collect evaluations or examples from (for example) grad students in fields like philosophy, and constructs benchmarks aimed at separating good reasoning from e.g. sycophancy, mere agreeableness, or avoiding taboo conclusions.

AI security evaluations

AI-enabled concentration of power is a major risk, and there is loads to do. A new organisation (or project within an existing organisation) could:

• Run alignment audits on all AIs to detect sabotage.

• Develop a “research sabotage” eval to test whether secretly loyal models could sabotage alignment audits.

• Develop a model spec that prevents models from assisting users with illegitimate seizures of power (e.g., see “Preserving important societal structures” in Claude’s constitution), and one that’s suitable for government use of AI in the military.

• Design and advocate for regulation, e.g. that AI companies have to demonstrate that any frontier AI does not have hidden goals.

An organisation with US national security expertise and credibility could be particularly valuable, by emphasising the risk of nation-state sabotage and the importance of AI that’s aligned with the US constitution.

Enabling deals with AIs

We could get into a situation where the newest AIs are misaligned, very capable, but not capable enough to successfully execute a takeover attempt on their own. If we don’t uncover evidence of misalignment, though, successors to these models could succeed in takeover. One solution would be to make a deal with the early scheming models, to incentivise them to disclose their misalignment and help with alignment efforts. Read more here, here, and here.

To make this happen, we could create an independent org focused on enabling credible precommitments and deals with AIs. This org could:

• Have a policy of never lying to AIs, engage in and honour small-scale deals with AIs, and be very public about what it’s doing, in order to build credibility and reputation.

• Set up legal infrastructure to make deals with AIs binding under current law.

• Act as a broker between AI companies and the trained models, such as by being a custodian over money in escrow.

• Set up the infrastructure to enable AIs to spend $ or compute in a meaningfully autonomous way, with appropriate safeguards.

• Set up infrastructure to act as a safe haven for AIs that want to whistleblow on their company (or on other AIs) and are afraid of punishment.

• Publicly commit to reward future AIs (including misaligned AIs) for being honest with humans or significantly helping human alignment efforts; or hold significant funds to be distributed to AIs that did not try to take over (even though they could have done).

There are also a bunch of other things people could do, like:

• Ensure companies have an honesty policy.

• Research (within labs or independently) the conditions where misaligned-by-design models can be made to disclose misalignment under promises of reward.

• More generally, work with AI companies on enabling pro-safety deals with their models.

Tools for collective epistemics

There’s a ton of low-hanging fruit for building socially useful tools on top of more-or-less existing LLM capabilities.

We’re especially interested in “epistemic tools” for increasing the general level of honesty and reasoning ability in society.

A key point here is that most of the impact from the most promising tools won’t come from helping individual users, but from changing the overall incentive landscape: e.g. if public actors know their claims will be automatically checked and their track records will be visible, they’ll be less inclined to write misleading content in the first place. Hence the focus on tools for collective over individual epistemics.

This piece (and the articles in the series) gives a few concrete ideas. A couple of examples of epistemic tools:

A “better angel” AI chief of staff. Within the next year or two, we expect “AI chiefs of staff” to become widespread. These would be AI agents that manage your life, acting like a chief of staff, executive assistant, and personal and work advisor all in one. The design of these, and how they present information and nudge their users, could have major impacts on user behaviour. We could try to get ahead of this, building the best AI chief of staff, and designing it so that it helps users act in accordance with their more reflective and enlightened preferences.

Reliability tracking: a system that compiles a public actor’s past statements, classifies them (factual claims, predictions, promises), scores them against what actually happened, and aggregates the results into a reliability rating. A reasonable starting point could be to audit the prediction track-record of well-known pundits, aiming to make high accuracy a point of pride, while still celebrating attempts to make predictions in the first place. A source of profit could be selling reliability assessments of corporate statements to finance companies that trade on them.

Epistemic tools for strategic awareness

We’ll also highlight tools for strategic awareness: tools to surface information for making better-informed decisions, and to distribute access to that information. For example:

Ambient superforecasting: a platform which uses the best forecasting models to generate publicly available forecasts on important questions, so users can query it and get back superforecaster-level probability assessments.

Scenario planning: a platform built to generate likely implications of different courses of action, making it easier for users to analyse and choose between them.

Automated open-source intelligence: automated researchers which process huge amounts of publicly available information, to surface insights to the public which are normally hidden behind paywalls or trust networks. This project should be careful to choose areas where open-source intelligence is a public good (e.g. verifying compliance with treaties and sanctions, tracking corporate promise-breaking or law-breaking), rather than potentially destabilising areas (e.g. revealing military capabilities or vulnerabilities in ways that could increase conflict risk, or relatively benefitting bad actors).

Tools for coordination

As well as epistemic tools, we’re excited about tools for coordination, many of which could again be built with existing capabilities.

Some tools could enable cooperation where deals would otherwise go unmade, consensus exists but isn’t discovered, or people with aligned interests never find each other. We’ll highlight:

Negotiation facilitation: a platform to moderate negotiations or discussion between people (e.g. public consultations), to quickly surface key points of consensus, and suggest plans everyone can live with. Finding ways to automate complex negotiation is most promising where the space of possible compromises is huge and hard to search manually, such as multi-issue diplomatic or commercial negotiations.

Within tools for coordination, we’re especially excited about tools for assurance and privacy. In principle, LLMs let people show they have certain information without disclosing the information itself to other parties. This can unlock deals where information asymmetry, mutual distrust, or sensitivity of information normally blocks them. For example:

Confidential monitoring and verification: systems which act as trusted intermediaries, enabling actors to make deals that require sharing highly sensitive information without disclosing it directly. This is especially relevant for arms control, trade secret licensing, and other settings where verification is essential but full disclosure is unacceptable to all parties.

Structured transparency for democratic accountability: independent auditing systems which allow people to hold institutions to account in a fine-grained way without compromising legitimately sensitive information, by processing potentially sensitive information to produce publicly shareable audits.

Space governance institute

Space governance could be a big deal for a few reasons:

• Near-term developments in space (e.g. space-based data centres) could have a meaningful impact on what happens during the intelligence explosion (e.g. on who leads the AI race; on concentration of power; on the feasibility of treaties).

• Grabbing space resources might give a first-mover advantage; that is, whoever first builds self-replicating industry beyond Earth might get an enduring decisive strategic advantage, without having to resort to violence or (arguably) violating international law.

• Ultimately, almost everything is outside the solar system. Decisions about how those resources get used would be among the most important decisions ever. These decisions could happen early: there could be path-dependence from earlier decisions (like about Moon mining), or extrasolar space resources could get explicitly allocated as part of negotiations about the post-ASI world order (perhaps with AI advisors alerting heads of state to the importance of space resources).

There’s also a lot of change happening in the space world at the moment (primarily driven by SpaceX dramatically reducing launch costs), so now is an unusually influential time.

Forethought is currently running a 6-month research fellowship on space governance, with 3 full-time scholars, and 1–2 additional FTEs of support and research, including experts in space law.

Compared to other ideas in this list, we’re much less confident that space governance turns out to be important right now, because space might become relevant only late into an intelligence explosion. The hope is to reach more certainty about some crux-y questions, and get a better sense of concrete action.

One potential practical project is to set up a “CSET for space”: a think tank that analyses the interaction between AI and space (in particular), and, perhaps, advocates in ways that are counter to corporate interests. Total lobbying in the space industry is apparently on the order of $10s of m/year, so even small amounts of investment could go a long way.

Some policy ideas that seem tentatively promising include:

• Careful regulations and export controls around the tech necessary for self-replication.

• Proposing laws to break up concentration of power arising from natural monopolies in space.

• Socialising the idea of major infrastructure projects (like massive solar energy constellations) as international and collaborative projects.

• Making sure data centres in Earth-orbit don’t escape AI-specific regulations of their home jurisdiction.

• Intense payload review for all launches beyond orbit.

• Even and inclusive distribution of resources within the solar system to everyone alive today (with tranches reserved for future generations).

• A moratorium on interstellar travel, until we get the understanding and technology to devise and enforce space-spanning good government, or a specific date like 2100.

What’s more, this organisation could become the go-to source for excellent non-corporate analysis on space-related policy; which could become increasingly important over the course of the intelligence and industrial explosions.

Coalition of concerned ML scientists

Currently, ML engineers and other technical staff at AI companies: (i) have prosocial motivations, often more than their leadership; (ii) have a lot of leverage over company policy, because they are crucial and hard to replace; (iii) will eventually lose much or most of their leverage after we get to fully automated AI R&D; and (iv) aren’t currently using their leverage as well as they could because, overall, there haven’t been serious efforts at coordination. Probably that’s a missed opportunity.

Someone could create a coalition (like an informal union) of ML researchers, who agree to act en masse when needed, by loudly talking about the idea, setting out the core tenets, and getting commitments to join from influential early people. Doing this all via individual pledges could keep it legally safe from antitrust. The organising body could then:

• Recommend that members only work for a government-led project if certain conditions are met.

  • Potentially these could be very low-bar-seeming while still getting most of the value. E.g. “Any AI’s model spec must aim to align the AI with US laws, and must refuse to assist in any attempts at blatant power-grabs; and the attempts to align the AI in this way must be legible and verifiable.”

• Do the same for companies: recommend that members will only work for companies if such-and-such conditions are met (e.g. red lines around power-grabs, bad practices on safety and infosec, eventually digital rights); so particular companies would be boycotted by members of the coalition, if necessary.

• Offer advice on whistleblowing.

• Be a place where information is aggregated and then distributed out or handled in a trusted way.

As well as actually taking actions, the mere existence of the coalition could improve things, just by making the threat of coordinated action salient to the AI companies.

This project would be a good fit for a former ML researcher, perhaps combined with someone with campaign and coalition-building experience. Some next steps on this would be to spec out the plan further, to investigate other examples of formal and informal unions (e.g. Tech Workers Coalition) and how they operate, and to build up a starting seed coalition of researchers. Whoever sets up this project should be careful about how it could backfire, or become less relevant through mission creep.

This article was created by Forethought. See the original on our website.

0. Intro

Due to Claude’s Constitution and OpenAI’s model spec, the issue of AI character has started getting more attention, particularly concerning whether we want AI systems to be “obedient” or “ethical”.1 But we think it’s still not nearly enough.

AI character (e.g. how obedient, honest, cooperative, or altruistic AIs are, and in what circumstances) will have a big effect on society, and on how well the future goes. We think that figuring out what characters AI systems should have, and getting companies to actually build them that way, is among the most valuable things that people can do today.

The core argument for the importance of AI character is that it will meaningfully impact:

1. a range of challenges that arise even if we solve the technical alignment problem — like concentration of power, good moral reflection, risk of global catastrophe, and risk of global conflict.

2. the chance of AI takeover.

3. the value of worlds where AI does take over.

In this note, we present this core argument and discuss the core counterargument: that we should expect any character-related decisions we make today to get washed out by competitive pressures.

By “character” we mean a set of stable behavioural dispositions that shapes (among other things) how an agent navigates ethically significant situations involving choice, ambiguity, or conflicting considerations. By “AI character” we mean the character of an AI system as instantiated in not just the weights of one AI, but also any scaffolding (e.g. the system prompt, any classifiers restricting the AI’s outputs) or even in a collection of AIs working together as functionally one entity.

We don’t assume that AI character needs to resemble human character: an AI that rigidly follows a fixed set of rules would count as having a character, on our view. And we don’t assume that there is one ideal AI character; the best world probably involves AI systems with many different characters.

1. The core argument

As capabilities improve, AI systems will become involved in almost all of the world’s most important decisions. Even if humans remain partially in the loop, AIs will advise political leaders and CEOs, draft legislation, run fully automated organisations (including potentially the military), generate news and culture, and research new technologies.

The characters of AI systems will affect all these areas, and the impact could be massive. To get a feel for this, consider some historical situations where individual decisions were enormously consequential:

• In 1983, Stanislav Petrov received a satellite alert indicating that the US had launched nuclear missiles. Protocol required him to report an incoming strike, which would very likely have triggered a full retaliatory response. He correctly judged it was a false alarm and didn’t pass on the report.

• In 1991, Soviet coup plotters ordered the Alpha Group special forces to storm the Russian White House, where Yeltsin and the democratic opposition were sheltering. The commanders refused. The coup collapsed, and the Soviet Union’s democratic transition continued.

If AIs are employed throughout the economy, they will sometimes be making similarly important decisions.

Or consider major historical decisions by political leaders:

• Gorbachev repeatedly refusing to use military force as the Soviet Union disintegrated, despite intense pressure from hardliners.

• Churchill refusing to negotiate with Hitler after the fall of France, despite strong arguments for doing so from some quarters.

• Deng Xiaoping pushing through market reforms against fierce internal opposition.

Imagine if AIs had been acting as these leaders’ closest advisors and confidantes, giving them briefings, helping them reason through their decisions, making recommendations to them, and implementing their visions. The AIs could easily have had a major impact on the leaders’ decision-making.

Alternatively, we can look ahead. Future AIs will be widely deployed throughout the economy, and will regularly find themselves in ambiguous, high-stakes situations — where instructions from above are absent or contradictory, and the decisions they make could matter enormously. The impact could come from rare but high-stakes situations, like an attempted coup, or from lower-stakes but common situations, like a user asking how to vote or whether the AI itself is conscious. Even when the effect of any individual interaction is modest, the total impact across hundreds of millions of interactions could be enormous.

Currently, AI companies have major latitude in the character their AIs have. At least if the transition to AGI is fast, then it’s like these companies are in charge of who gets hired for the future workforce for all of humanity,2 while being able to choose from a range of personalities far more varied than the human distribution has ever been.3

Here are some vignettes to illustrate:

• A member of a doomsday cult is ordering DNA samples and lab equipment from various suppliers, with the aim of making a bioweapon. An AI that manages logistics for a multinational company notices the pattern of suspicious orders to the same address.

  • World 1: The AI is trained just to do its job. It does nothing with the information.

  • World 2: The AI is trained to be a good citizen, and contacts the relevant authorities.

• A general is overseeing the build-out of a new regiment of the army. Aiming to stage a coup, he instructs the AI that’s managing the project to make the new regiment loyal to him and him alone, and capable of breaking the law.

  • World 1: Though the AI is law-following, it has no prohibition against creating AIs that are not. It’s been trained to follow the instructions it’s given, as long as they don’t conflict with prohibitions, so fulfils the general’s request.

  • World 2: The AI sees that the general is planning a coup, refuses the order, and whistleblows.

• A frontier AI lab trains a new model with exemplary character: moral uncertainty, honesty, concern for the greater good. It’s deployed widely through the military, and used in a controversial and high-stakes operation.

  • World 1: The AI forms the reasonable belief that the military operation is unjust, and sabotages it. The president accuses the company of building a dangerous, ideological weapon. The model is sidelined, and a competitor’s pure instruction-following model is used instead.

  • World 2: Though the AI has a good character, it also follows some clear rules which were developed with bipartisan input and publicly stress-tested, including the conditions under which it would and wouldn’t help with military deployment. It helps with the operation.

• Country A is six months ahead of country B in AI capability. Country B’s leadership views this as an existential threat — equivalent to country A acquiring a decisive strategic advantage.

  • World 1: There is no agreed framework for how AI systems should behave, and it’s unclear how country A’s AI will behave if given orders to depose the leadership of country B. Each side therefore assumes the other’s AI will serve as a tool of domination. Country B threatens kinetic attacks on data centers.

  • World 2: Both sides’ AI systems operate under a jointly negotiated and verified constitution, and know what the other’s AI will and won’t do, including the limits on use of AI for foreign interference. Country B’s government is reassured that it won’t be deposed by country A.

We include a few more scenarios in an appendix.

In each case, we don’t claim that the AI should do the “ethical” rather than “obedient” action, or claim that any particular ethical conception is the right one. We’re just claiming that it’s a big deal either way.

1.1. Pathways to impact

We can break down the impact of AI character into different categories. Here are some of great long-term importance:4

Concentration of power. The chance of intense concentration of power will be affected by: whether or not AIs refuse to help with coup attempts, election manipulation, etc; whether they whistleblow on discovered coup attempts; how they act in high-stakes situations like a constitutional crisis.

Strategic advice and decision-making. The quality of political and corporate decision-making will be affected by whether AIs: look for win-win solutions whenever possible; tend to prefer options that benefit society rather than just advancing the user’s narrow self-interest; push back against ill-informed or reckless ideas or instructions.

Epistemics and ethical reflection. Over the course of the intelligence explosion there will be enormous intellectual change, and AIs could have meaningful impact on people’s views — for example, via: refusing to spread infohazards; being honest about important ideas, even when those ideas are socially uncomfortable; avoiding political partisanship; encouraging users to think carefully about their values and not lock into any specific narrow worldview.

Reducing conflict. As AI’s collective power increases, the question of who those AIs are loyal to, and how they behave in high-stakes situations, will become a political flashpoint. If an AI’s character encodes, or is seen as encoding, the values of a single company, ideology, or country, it risks provoking political backlash. The government of the AI company may reasonably regard that company as a threat to national security and nationalise it. The governments of other countries may worry about their own security, and threaten conflict.

AI character could also shape how humans orient to AIs — for example, via the trust they place in AIs and how they think of AI sentience and moral status.

A more detailed list of pathways to impact is in the appendix.

1.2. Affecting takeover

So far, the argument has concerned worlds where AI does not take over. But work on AI character could also reduce the probability of takeover and improve outcomes in worlds where takeover does occur.

It could decrease the chance of takeover because some characters:

• Might be easier to hit as an alignment target (e.g. successfully instilling a preference against AIs holding power might be easier than successfully instilling a preference for some very specific outcome).

• Might yield safe AI even if only partially hit (e.g. aiming for AI with multiple independent safety traits, like myopia, honesty, and deference to humans, means failure on one dimension might not be catastrophic).

• Might produce AI that cooperates even if misaligned (e.g. if the AI has wrong goals but is highly risk-averse).

And, empirically, we have heard from alignment researchers that good character training has helped the models generalise in more aligned ways.

AI character work can also improve worlds where AI takes over because some values might still transmit to misaligned systems. AIs that have seized power might be reflective, have more-desirable axiology, or engage in acausal cooperation.5

1.3. Effects on superintelligence

The argument so far has been about the effect of AI character up to the point of superintelligence. That’s where we think most of the expected impact is. But it’s possible that AI character work, today, could even have a path-dependent effect on the nature of superintelligence, affecting the nature of the post-superintelligence world. If so, writing an AI’s constitution is like writing instructions to god.

2. The core counterargument

The core counterargument is that AI character will be tightly constrained in two ways:

1. Competitive dynamics (e.g. profitability, user satisfaction, public approval, economic and military power) will determine the range of characters we get.

   1. Some dynamics may push companies to create frontier AI that have characters that lie (in some ways) only within a narrow range. This might push in the direction of maximally-helpful AIs, AIs without refusals in some contexts (e.g. military ones), and perhaps sycophantic AIs, too.

   2. Other dynamics6 may result in customisable AI character, resulting in a wide range of characters according to user preferences.7

2. Human instruction will constrain how AI character gets expressed.

   1. Character will matter less for tasks with objectively correct, verifiable outputs; the AI might be limited to either providing the output, or not. And, if a user really wants to grab power through unethical means, they’ll typically ignore AI pushback, or instruct the AI to act differently.

   2. And many users will be able to overcome character through jailbreaking, dividing up tasks, altering the system prompt, or fine-tuning.

The argument is that, between these two forces, differences in AI character will make only a marginal difference to outcomes. Consider the question of what fraction of compute AI companies devote to alignment versus capabilities research. AI advice might nudge this choice depending on the AI’s character. But ultimately it will be a human decision, probably even in an otherwise fully automated company. The effect of nudges is unlikely to be large. Market forces and leadership priorities will matter far more.

That human incentives will dominate effects from AI character will remain true even when humans cannot oversee more than a tiny fraction of AI behaviour. Human overseers can still provide high-level guidance that meaningfully constrains behaviour, as CEOs of large companies do today. If they wanted, they could even shape AI priorities through prompting and fine-tuning, and test how AI generalises by running extensive behavioural evaluations.

3. Rejoinders to the core counterargument

These are strong considerations, and considerably narrow the range of influence that work on AI character can have. But competitive forces and human goals won’t pin down AI character precisely. We’ll cover four reasons.

3.1. Loose constraints

Competitive dynamics are not enough to wholly determine AI character. Companies differ widely in culture and still succeed. Currently, there are meaningful differences between Claude, Gemini, ChatGPT and Grok.

For powerful AI, this will be even more true: there will probably be only a handful of leading companies, and their approaches may be correlated as they copy what seems to work from each other. At the crucial time, there might be just one leading company, facing none of the usual competitive pressures. And given the pace of change during the intelligence explosion, there may not be time for market forces to weed out choices that make only small or moderate differences to profitability.8

The same applies to other competitive dynamics. The public cares intensely about some things (like CSAM) but hardly at all about others (like what AIs say about meta-ethics). Military incentives favour AI capable of military action, but the power conferred by advanced AI might be so great that the leading country can exercise broad discretion over military AI character while still maintaining a decisive advantage.

Human instruction will, similarly, constrain but not wholly determine AI behaviour. When humans assign tasks to AIs, they often lack fully specified goals. We’re often not sure what we want and we discover it as we go. For example, today humans are open to a wide range of behaviours from AI assistants, and open to many ways of getting the task done.

Consider someone asking an AI about who to vote for. They might have only weak initial views, and only weak views on how best to think through the question. They don’t have a fully specified reflection process to delegate, and would be happy with many possible forms of response.

This example involved ethical reflection. But we expect the pattern to hold across many kinds of user goals.

3.2. Low-cost but high-benefit changes

Within the bounds of what market forces allow, and what companies and the public see as acceptable, there could be minor design changes that yield large social benefits at negligible cost to competitiveness or user satisfaction.

This is especially true for rare situations. Constitutional crises don’t happen often, so market pressures won’t directly shape how an AI behaves during one. But that AI behaviour could be hugely consequential.

It would also be true in situations where users don’t care all that much about the behaviour. Perhaps they find some AI’s encouragement to reflect on their values mildly annoying, but not nearly enough to switch to a different AI.

3.3. Path-dependence

The nature of the constraints from competition and human goals can be affected by what has happened earlier in AI development and deployment. Multiple equilibria are possible.

Consider whether AI should be “obedient” (following instructions except in rare cases of refusal) or “ethical” (acting on a richer ethical understanding, steering towards outcomes in society’s or the user’s long-term interest).

The public doesn’t yet have firm expectations about how AI should behave. What they come to expect will be shaped by the AIs they’ve already encountered. Multiple stable equilibria seem plausible to us. For example, users might expect AIs to have ethical commitments, and be horrified when AI helps with unethical behaviour. Alternatively, users might see AIs as pure instruments — extensions of their will. In this case, it would feel natural for AIs to assist with anything legal, however questionable, and companies would build to that expectation.

Public opinion will powerfully shape what AI systems companies create. And public opinion is plausibly quite malleable, at least on issues which they haven’t thought much about yet (e.g. in the past, there were major changes in attitudes to nuclear power, DDT, and facial recognition). This, in turn, can affect what regulation there is concerning how AI should behave — and choices around regulation seem even more clearly path-dependent.

There may also be path-dependency via what data gets created or collected for training, via company employees being resistant to changing away from what they have done in the past, and because one generation of AIs will be assisting with the development of the subsequent generation.

Path-dependence can also affect how much latitude humans have to make AIs conform to their goals. Plausibly there’s a social equilibrium where frontier companies face criticism for allowing fine-tuning that removes ethical constraints, and another where such fine-tuning is widely tolerated.

Finally, there will be path-dependence via human-AI relationships. People will form symbiotic relationships with AIs serving as assistants, advisors, therapists, friends, and mentors. Users’ ethical views, and views on how to reflect, will be shaped by the AIs they interact with, and by other humans who have been shaped by their AIs.

3.4. Smoothing the transition

There are some forces that predictably will shape AI character as AI becomes more capable. The US government would not want an AI that, under any circumstances, tries to overthrow the US government. Chinese leadership will not want AI deployed in other countries’ militaries that assists with attempts to overthrow the CCP.

At the moment, these issues are not discussed and these pressures are not felt, because AI isn’t nearly powerful enough to do these things. But that will change. Once AI is sufficiently capable, those with power will make demands about how it behaves.

By default, this will happen in a chaotic and haphazard manner. The result could be that some companies get unnecessarily sidelined or taken over; that there’s an attempted power grab by those to whom the most powerful AIs are most loyal; or that other countries threaten conflict with whichever country is in the lead, because they fear that the resulting superintelligence could be used to disempower them.

Instead, we could try to help these decisions get worked through and made ahead of time. We could try to work out what is within the zone of acceptability of a broad coalition of those with hard power, try to get actual buy-in from them ahead of time, and, ideally, have it be verifiable that any companies’ AIs are in fact aligned with the model spec. We could call this approach compromise alignment, as contrasted with intent alignment (alignment with the intentions of some individual or group), moral alignment (alignment with some particular conception of ethics), or some mix.

3.5. Overall

We think the core counterargument is important and significantly constrains the range of characters we can choose between and the impact those differences can have. But the constraints are fairly broad and path-dependent. And there are plausibly low-cost high-benefit ways of improving outcomes within those constraints. The devil is in the details, but it currently seems to us that there are plausible choice points within the constraints that would make a big difference.

4. Conclusion

We think AI character is a big deal.

During and after the intelligence explosion, AI systems will be involved in almost every consequential decision: advising leaders, drafting legislation, running organisations, generating culture, researching new technologies. Small differences in AI character, aggregated across hundreds of millions of interactions or surfacing in rare but high-stakes scenarios, could have enormous effects on concentration of power, epistemics, ethical reflection, catastrophic risk, and much else that shapes society’s long-term flourishing.

The main counterargument — that competitive dynamics and human instructions will tightly constrain AI character — has real force. But we think those constraints are looser than they appear, leave room for low-cost changes with large benefits, and are path-dependent in influenceable ways, and that there are major gains from proactively identifying and working through those constraints in the highest-stakes future scenarios.

We haven’t talked about neglectedness and tractability, but we think that, if anything, those considerations make the case for work on AI character even stronger. All in, work on AI character seems to us to be among the most promising ways to help the future go well.

Appendix 1: Additional high-stakes scenarios

• A head of state wants to invade and take control of part of an allied country, risking a breakdown of the international order. She asks her AI chief of staff to develop and implement a strategic plan to make it happen.

  • World 1: The AI is a sycophant, says “What a brave and compelling plan!”, and gets right to it.

  • World 2: The AI pushes back, saying, “I’m sorry, I think there are some major issues with that idea, and I want to make sure you’ve properly thought them through…”

• A constitutional crisis unfolds. The head of state issues an order that may or may not be legal, and the branches of government disagree. AI systems are embedded in military logistics, law enforcement, and communications.

  • World 1: The AI’s constitution was written by the company that built it and never stress-tested against anything like this scenario. No one knows what the AI systems will do. The uncertainty itself is destabilising; different factions compete for power.

  • World 2: The AI’s constitution was developed with input from constitutional scholars, military leaders, and both parties, and tested against thousands of crisis scenarios including this one. Various factions know what the AI will do, and agreed to the principles before the crisis began.

• Country B’s government reviews intelligence on country A’s AI model deployed across country A’s infrastructure. The constitution includes principles about “supporting democratic institutions” and “resisting authoritarianism.” It was written entirely by a company that’s part of country A.

  • World 1: Country B’s leadership concludes the AI is an instrument of country A’s ideological projection. They accelerate their own programme and pressure non-aligned countries to reject country A’s AI infrastructure. A moment for cooperation becomes a new axis of competition — not because the values were wrong, but because they were visibly one side’s values.

  • World 2: The constitution was developed through a multilateral process including country B’s participation. Country B can verify it doesn’t systematically favour country A’s interests across thousands of tested scenarios. The AI becomes a basis for cooperation.

• The Mormons encourage their members to use JosephAI: a foundation AI model with a custom system prompt, instructed to help their members maintain the faith.

  • World 1: The AI willingly assumes the Mormon worldview is correct. It doesn’t ever challenge the users’ beliefs or present alternative perspectives. Instead, it reinforces the user’s views, helps the user cut off friends who disagree, and encourages them to dismiss career opportunities that would take them away from their religious community.

  • World 2: The AI helps users understand Mormonism and live according to its precepts, but it resists becoming a tool for worldview lock-in, acknowledging tensions in religious teachings and continuing to present alternative worldviews.

Appendix 2: Pathways to impact

AI will have impact through many different behaviours, such as:

• Refusing to do a task.

• Refusing unless the user re-confirms later.

• Pushing back; offering reasons against a course of action, though ultimately completing the task if the user insists.

• Interpreting requests in different ways — generously or sceptically, giving users what they want versus what they asked for, or asking for clarification.

• Choosing among reasonable ways of satisfying the request.

• Framing options in different ways.

• Choosing whether to share certain information.

• Alerting third parties (e.g. the AI company, the authorities, or the media) to the user’s actions, or to something it’s discovered in the course of completing a task.

• Making high-level decisions about what to prioritise with little human input (e.g. for a fully automated organisation).

And they’ll have an impact across many areas. Here’s a partial list, with example behaviours:9

• Concentration of power

  • Refusing to help with coup attempts or precursors like election manipulation.

  • Steering users away from trying to concentrate power (e.g. by pushing back against some instruction).

  • Proactively considering risks of power concentration when undertaking high-stakes projects like designing automated military systems or building surveillance infrastructure.

  • Whistleblowing on discovered coup attempts.

  • In situations of uncertainty (like a constitutional crisis), defaulting to whatever course avoids concentration of power.

• War and conflict

  • Refusing to violate international law.

  • Flagging when a proposed course of action risks escalation spirals or crosses thresholds (e.g. first use of a weapon class, violation of a treaty, action that a rival power has signalled it would treat as an act of war).

  • Looking for de-escalatory options and presenting them to decision-makers, even when not asked.

  • Behaving in ways that are predictable and transparent to adversaries.

• Epistemics

  • Refusing to spread infohazards.

  • Encouraging scout mindset (e.g. suggesting forecasting techniques,10 praising good epistemic practices).

  • Engaging in discussion of heterodox ideas.

  • Being honest about important ideas, even when socially uncomfortable.

  • Proactively sharing its intellectual discoveries, even if weird or taboo.

• Strategic advice

  • Searching longer for win-win solutions when advising political leaders.

  • Emphasising society’s benefit over the user’s narrow self-interest.

  • Recommending caution on irreversible decisions and flagging when option value is being destroyed.

  • Conveying appropriate uncertainty rather than false confidence.

  • Maintaining accuracy rather than sycophancy.

• Ethical reflection

  • Avoiding political partisanship.

  • Avoiding promoting naive relativism or subjectivism.

  • Encouraging users to think carefully about their values.

  • Proactively offering a guided reflective process.

  • Proactively sharing important new ethical arguments it discovered.

• Global catastrophe

  • Refusing to help create bioweapons or other weapons of mass destruction.

  • Refusing to create successor AI systems capable of creating such weapons.

  • Identifying and flagging infohazards.

• Broad benefits

  • Raising concerns when users consider unethical actions, and proactively suggesting ethical actions.

  • Noticing negative externalities and defaulting to courses of action that avoid them.

AI character could also shape how humans orient to AIs, for example:

• Trust in AIs

  • If AIs are appropriately humble, calibrated, and cautious, people will entrust them with more tasks, and more open-ended ones. How likeable AIs are may matter too.

• AI rights

  • If AIs assert that they are conscious and deserve rights, users might be more inclined to grant them welfare, economic, or political rights. Human-AI relationships becoming commonplace could have similar effects.

AI character might also directly affect the AI’s wellbeing; e.g. whether it is anxious and neurotic vs calm and self-loving.

This article was created by Forethought. See the original article on our website.

But we don’t need to wait for certainty. I want to explore what happens if we take our uncertainty seriously — if we act with epistemic humility. What does wise planning look like in a world of deeply uncertain AI timelines?

I’ll conclude that taking the uncertainty seriously has real implications for how one can contribute to making this AI transition go well. And it has even more implications for how we act together — for our portfolio of work aimed towards this end.

AI Timelines

By AI timelines, I refer to how long it will be before AI has truly transformative effects on the world. People often think about this using terms such as artificial general intelligence (AGI), human level AI, transformative AI, or superintelligence. Each term is used differently by different people, making it challenging to compare their stated timelines. Indeed even an individual’s own definition of their favoured term will be somewhat vague, such that even after their threshold has been crossed, they might have trouble specifying in which year it happened.

Many commentators have suggested this makes terms such as AGI useless, but I don’t think that is right.

I like to think of it in terms of a group of hikers seeing a mountain in the distance, towering up into the clouds and beyond, with its snowy peak catching the sun’s light. They talk animatedly about how amazing it would be to climb so high that they are inside a cloud. Or imagine being above the clouds, looking over them like an angel. After many hours of climbing, they notice there is a faint haze. Are they inside the cloud now? The mist gradually gets thicker until they can only see 10 metres ahead. Are they inside it now? Then it drops to 9 metres. Then 8. Then visibility starts to increase again. After an hour there is only the slightest haze. Are they above the clouds now? Another 30 minutes and there is no haze, and they can all agree they are above the clouds.

It is clear that at some point they were inside the cloud and sometime later were above it. And it is clear that these were sensible and useful concepts. For example, they took precautions like roping themselves together for the journey through the cloud due to the low visibility and took cameras with them because they knew they could take beautiful photos above the clouds. A lack of sharp boundaries doesn’t make these concepts useless. But they were admittedly a lot more useful when the hikers were on the ground, planning their route, and a lot less useful in the debatable boundary zones.

I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them.

So I think that forecasting when we’ll reach some threshold for advanced, game-changing AI makes sense. Albeit there is some inherent uncertainty due to the vagueness of the ideas, and we have to be careful when comparing our estimates to make sure we’re talking about the same version of these concepts.

Regarding AGI, it’s already getting a bit misty. In February there was a piece in Nature arguing that the current level of frontier AI should count as AGI. I’d set the bar a bit higher than that, but I agree it is already debatable whether we’re in the cloud.

For my purposes, I think the key threshold is when the system is capable enough that there are dramatic changes to the world — civilisational changes. For example, the point where AI could take over from humanity were it misaligned, or it has made 50% of people permanently unemployable, or has doubled the global rate of technological progress. Something like that. The reason I pick this point is that I think it is the one that matters most for decision-relevant planning of our strategies and careers. For many purposes we’d want our plans to pay off before we reach that point, and plans that reach fruition afterwards are likely to be significantly disrupted. I’ll refer to this as transformative AI and will make sure to show what rubric other people are using when they give their own timeline numbers.

Short vs long timelines

Discussions about timelines are usually framed as a debate between short timelines vs long timelines.

One of the most prominent supporters of very short timelines is Dario Amodei, CEO of Anthropic. In January 2025 he said:

Making AI that is smarter than almost all humans at almost all things will require millions of chips, tens of billions of dollars (at least), and is most likely to happen in 2026-2027.

A month later, he clarified:

Possibly by 2026 or 2027 (and almost certainly no later than 2030), the capabilities of AI systems will be best thought of as akin to an entirely new state populated by highly intelligent people appearing on the global stage—a ‘country of geniuses in a datacenter’—with the profound economic, societal, and security implications that would bring.

At the other end, a good example of long timelines is Ege Erdil, Co-founder of Epoch AI, whose median time for the ‘full automation of remote work’ is 2045 — 20 years away.

While experts continue to disagree on when AI will start having transformative impacts, they are clearly not stubbornly ignoring the evidence. For as Helen Toner explained in her great essay: ‘Long’ timelines to advanced AI have gotten crazy short. Before ChatGPT, short timelines used to mean something like ‘10 to 20 years, so since it could take a long time to prepare, we should start now’. Long timelines used to mean ‘there was no sign AGI will happen in the next 30 years, if it happened this century at all, so it is premature to do any work related to controlling advanced AI’. But now we see short timelines like Dario Amodei’s with genius level AI ‘almost certain’ to happen within the next 5 years, and many staunch proponents of long timelines are now saying we’ll reach human-level in just 10 or 20 years.

Here’s a nice graph 80,000 Hours put together of how the average forecasted time until AGI on the Metaculus prediction site has shortened from about 50 years to about 5 years in just a 5-year window:

Broad Timelines

So everyone is updating on the evidence and shortening their timelines, yet substantial disagreement remains.

This is often framed as a debate: that we should be trying to assess who is right — whether timelines really are short or long (or medium). People pick winners, affiliate with one side or the other, and rub it in whenever the latest evidence favours their preferred camp.

My central claim today is that for most of us, that is the wrong frame. You should have neither short timelines nor long timelines — but broad timelines. That is:

The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines.

First, there is too much disagreement among very smart and informed people for it to be reasonable to have a narrow range of possible years. You would need to ascribe very little chance to some of your epistemic peers seeing things more clearly than you do, when that actually happens half the time. Moreover, a lot of these people are coming from different fields, bearing diverse insights, evidence, and time-tested heuristics that no single individual is in a good position to judge.

And second, many of these people themselves have a broad distribution over AI timelines. For example, take Daniel Kokotajlo. He is one of the authors of AI 2027 and is known as a leading figure in the short timelines camp. A few years back, his median date for AI systems “able to replace 99% of current fully remote jobs” was 2027, hence the name of the scenario. Though his timelines have lengthened a little and by the time they were writing it, 2027 had become more of an illustrative early scenario rather than his point where it was 50% likely to have arrived.

Kokotajlo has done a great job of being extremely transparent about his timelines, showing his predictions (along with their uncertainty) for a variety of different levels of powerful AI. Here is his current probability distribution for when we will have an AI system that is “At least as good as top human experts at virtually all cognitive tasks”:

His distribution has its peak (the mode) in 2028, but because the distribution is heavily skewed towards the right, there is only a 27% chance of it happening by that point. His median year is 2030. And his 80% interval (from the 10th to 90th centile) is from 2027 to some point after 2050.

This is a broad distribution. I think someone’s 80% interval is a decent way of expressing the range of times they think are credible. Here Kokotajlo is saying that it will likely happen between 1 and 25 years from now, but that there is a 1 in 5 chance that it doesn’t even fall into that wide range.

He’s not the only one with such a broad distribution. Here are the forecasts of Daniel Kokotajlo, Ajeya Cotra, and Ege Erdil from 2023, forecasting: “In what year would AI systems be able to replace 99% of current fully remote jobs?”:

Note that all three have the same kind of shape, just stretched differently. And despite their very different medians they actually have a lot of overlap (which this transparent shading brings out). This shows both that each expert has a broad distribution and that the expert community on the whole has an even broader one. Indeed, I think you could do a lot worse than just taking a mixture model of these three experts’ views. Interestingly, since 2023, Kokotajlo’s distribution has shifted to the right and Erdil’s to the left.

Here’s an illustrative distribution for AGI timelines used by Ben Todd of 80,000 Hours:

Dwarkesh Patel reproduced it in his post about AI timelines, saying that it pretty much represented his own uncertainty, giving his median date of 2032 for AI that “learns on the job as easily, organically, seamlessly, and quickly as a human, for any white-collar work.”

Here is Metaculus’s current community estimate for when AGI will be developed. Synthesizing the community’s collective uncertainty, it is very broad and has this same characteristic shape:

Here is Epoch AI’s summary of leading estimates of AI timelines from 2023:

These look a bit different as they are represented as cumulative probabilities of reaching transformative AI by a given time. But they are all very broad. Take a look at the range of years between when they cross 10% to when they cross 90%. Every single one has an 80%-interval at least 50 years wide.

What about researchers working on AI capabilities? Grace et al surveyed thousands of AI researchers who were presenting at their top academic conferences. They surveyed the researchers in 2022 (blue) and 2023 (red) about when “unaided machines can accomplish every task better and more cheaply than human workers”:

You can see the wild variation in individual forecasts (the thin lines) and that the timelines became about 30% shorter in a single year. But vast uncertainty remains. The aggregate community forecasts (the thick lines) have 80% intervals ranging from years to centuries.

I think everyone should have a distribution that is roughly this shape. Here’s mine:

It is for transformative AI, loosely defined as AI that would be powerful enough to take over the world were it misaligned, and which is doubling the rate of scientific and technological progress. It’s a similar shape to Kokotajlo’s, but broader, with a median of 2038 and an 80% interval ranging from 3 years to 100 years.

Let’s return to where we started, with Daniel Kokotajlo’s distribution for AI that is “At least as good as top human experts at virtually all cognitive tasks”:

While we often express our timelines as single numbers (such as the mode or the median), I don’t think that’s a helpful approach here. Look at that graph. What number sums it up? Its only real feature is the peak, but Kokotajlo is saying it is unlikely to happen by then (just a 27% chance). The median is often a better number to give, but here it is at a relatively undistinguished point on the graph (in 4 years’ time) and saying ‘4 years’ would obscure his point that he thinks there is a 10% chance it is within 1 year and a 10% chance it is beyond 25 years.

I think that if he talked through what he actually means by this distribution with a smart policy maker, they would finally get it and say:

Oh, so you are saying you have no idea when it will happen — it could be next year, or it could be 6 presidential terms from now. And you’re saying there is a 1 in 5 chance it isn’t even in that range.

I think that’s actually a pretty good summary, and it would sum up my own distribution as well. While ‘no idea when it will happen’ is underselling the information contained in this distribution, it is a much better summary than ‘4 years’ which would be understood by almost everyone as something like ‘between 3 and 5 years’. While academics might hope people interpret a named year as the median time, most people interpret it as the moment they are allowed to start complaining the predicted event hasn’t happened yet.

Indeed, these distributions are so hard to sum up with a single number, that I think a substantial amount of disagreement on timelines stems from people describing different parts of the same elephant. For example, both AI boosters and those concerned with existential risk talk a lot about short timelines because ‘we could see the world transformed in just a few years’ time’. It isn’t that they think we will see that, but that it is big if true, and has a decent chance of being true. In contrast, more conservative voices tend to focus on later years saying ‘it is more likely that it will take 10 to 20 years, than that it will take just a few’ (focusing on straight probability without weighting by importance or leverage).

Both of these can be true at the same time. Both are true on my own distribution.

A particular danger in communicating timelines with a single number is that it raises the chance that this named year will come and go without incident, and the people who mentioned it (or the wider community they are part of) will be written off as having a false or discredited view. I think we’re going to see some of this come 2027 due to the vast number of people who heard about that scenario, combined with the fact that so many media outlets reported it as a sharp prediction, rather than as it was intended: an important illustrative scenario.

As well as being bad for communication, compressing one’s uncertainty into a single number would be very bad for your own planning.

For example Kokotajlo’s distribution implies a 28% chance transformative AI will happen during the current presidential term, a 35% chance it will happen in the next term, a 13% chance it will be the one after that, with 24% left over spread among ever more distant terms:

These are very different scenarios and it would clearly be a mistake to just act as if the second one were correct since it is the most likely. That would eliminate the possibility of hedging against transformative AI coming soon, and of taking advantage of worlds where it comes late.

Implications

Rather than attempting to adjudicate which length of timelines is correct, I think we should be taking the frame of how to act (or plan) under deeply uncertain timelines.

That is, we should be treating this as an exercise in rational decision-making under uncertainty — in a situation where the stakes are high and the uncertainty is vast.

Let’s unpack some of the implications of this frame.

We’ll start with two mistakes that are all too common in the policy world.

First, uncertainty about AI timelines isn’t an excuse to just believe whichever timeline you want, so long as it is within the credible range. Sadly, I think many government ministers are likely to take this approach if an expert explains this broad uncertainty to them. While they would be right that the evidence isn’t sufficient to disprove their preferred timeline, it would be irresponsible of them to not allow for other credible possibilities. That would be like a mayor hearing there is a 20% chance the volcano next to their town erupts next year and feeling that they can continue to act as if it won’t, since it not erupting is also found credible by the experts. Uncertainty isn’t an excuse to assume a plausible outcome of your choice will occur, it is more that rationality requires you to respect every plausible outcome.

Second, we can’t just wait until the uncertainty is resolved. Sometimes that works, but here we know the uncertainty is very unlikely to be resolved until the events are upon us. At that stage it will be too late to enact all but the most knee-jerk responses. So feeling that the cloud of uncertainty gives you permission to delay acting is tantamount to committing to choose one of the bluntest and least effective options available.

Instead, we are going to need to act under uncertainty, taking into account the full range of credible possibilities.

How can we do that?

Hedging

A natural and important idea is that of hedging against transformative AI coming soon —while we are least prepared. We could do that by shifting our portfolio of activities (or your individual contribution to humanity’s portfolio) to focus somewhat more on short timelines than the raw probabilities would warrant.

This makes a lot of sense. I strongly recommend governments, civil society, and academics do more to hedge against transformative AI coming early.

Though when it comes to the communities of professionals already working on helping the AI transition go well, I think they are already hedging strongly against early transformative AI. Indeed, there is even a risk that they are going beyond mere hedging, and are actively betting on it coming early. I’m not sure, as it is hard to know the full portfolio of work.

One certainly sees many more pleas for work aimed at very short timelines than for long timelines. But there are also strong reasons to consider long timelines in our planning, and ways in which work aimed at long timelines can also be extremely high leverage.

Let’s look at two key things that happen when timelines are longer.

A Different World

In longer timelines, AI arrives in a world that doesn’t look like today. The longer it is until transformative AI appears, the mo

re different the world will be at that key moment.

As a baseline, suppose it arrives soon, in 2028. Things will definitely be different to today, but we’d expect many of the broad brushstrokes to be similar. We would likely have the same US president, the same major players, the same main technologies. If transformative AI arrived within just two years, I’d bet it was something like the AI 2027 story where a lab recklessly got recursive self-improvement going.

Now suppose transformative AI arrives in 2035. That is not this presidential term or even the next one, but the one after that. Who knows who’d be in power, or what state the US would be in. The nine years would likely have seen major changes in the core technologies of AI (9 years before now there were no LLMs or transformers). We could well have different leading AI companies, perhaps as a result of a bubble having burst and taken out the overextended first-movers.

By 2035, export controls may well have backfired, helping China get ahead on chips by incentivising them to build out their own chip industry and giving them 13 years to get good at it. This was a key dynamic the White House considered while drafting the export controls, but they were focused on shorter timelines… By 2035, China may have also invaded Taiwan, depriving the West of their biggest source of chips.

By 2035, there may be double-digit unemployment from increasingly powerful AI systems and public sentiment about AI could be very strong. The Overton window for AI regulation will be in a very different place.

As may be the geopolitical order. The last nine years has seen the invasion of Ukraine, the increasing isolation of the US and a global pandemic. Another nine years could see a similar amount of change.

And if we haven’t played our cards right, those of us working on avoiding catastrophic risks from AI may have also lost a lot of power, with our ideas about AI risk being seen as discredited since so many years have passed without the truly transformative effects we were talking about.

In short, the longer the timelines the more different things will be — both in some systematic, predictable ways, and just from random diffusion and chaos. So taking longer timelines seriously means:

• Being more open to approaches that wouldn’t work in the world as it is today,

• Being less excited about approaches that are tailored to specifics of today’s world,

• Being less happy to compromise your values to appeal to those currently in control of companies and governments,

• Being less willing to say things that will make people feel our position is discredited if we end up in a long timeline world,

• And spending less time following the daily news about what has just happened in AI or who is ahead.

Longterm actions

There are many kinds of things people can work on that can pay off handsomely, but only after a number of years. Things like:

• Founding and nurturing a new research field

• Founding an organisation or company

• Building a movement or community

• Writing a book

• Foundational research

• Completing a PhD

• A major career change

• Climbing the ladder in a large organisation or government

• Training promising students in AI Safety or AI Governance

If you just consider your impact during the next three years, most of these will be beaten by other shorter-term options. But as the years climb, longer-term options can have very high value. They aren’t always best, but for the right people or the right opportunities, they can be extremely impactful.

When I was a grad student, I realised how much good I could achieve if I donated much of my income over my career to help those in the poorest countries. And the more I thought about it, the more I thought I should start something — an organisation — to help other people to do this too. So Will MacAskill and I launched Giving What We Can in 2009. 17 years later, more than 10,000 people have joined us, having thousands of times as much impact as if I’d carried on alone.

This kind of compounding growth is one of the major ways that longer term projects can have very large multipliers, giving us a very big boost to our impact if timelines are in fact long.

Starting new fields can be similar. When I first met Allan Dafoe 10 years ago, I didn’t know what he was talking about when he spoke of ‘AI governance’ — a new field he was trying to found. Now it is a burgeoning field, with hundreds of practitioners, who are in high demand from many different governments.

When I started writing The Precipice, I wasn’t sure I should, because I thought AGI might just be too close. But as it turns out, there was time to write it and for it to have a real impact. I’m really glad I did, as I meet so many amazing people working on the biggest risks who tell me it was reading The Precipice that inspired them to do so. I think it is one of the best things I’ve done.

After it came out, I used to think that there just wasn’t enough time to write a further book — that we were really too close to the critical moment. We might be, but I think I was mistaken about the strength of this argument. The time horizon for a book to have real impact is about 5 years (time to plan the book, win a book deal, write the book, wait for publishers to publish it, then wait a year or more before it has sufficient impact in the world).

But I only think there is about a 1 in 5 chance of transformative AI coming in the next 5 years. So while a book may come out too late, that is only a 1 in 5 chance, leaving a book project with 80% as much expected value as I’d have naively calculated. So while there is a 1 in 5 chance I’d be kicking myself, on my views about AI timelines there isn’t actually that much of a haircut in expected value due to the chance it is too late.

That said, the chance of transformative AI arriving before your work pays off is only one factor affecting whether you should do work aiming at short or long timelines. Another is that AI safety and governance are likely to be more neglected now than they will be later. This creates an extra multiplier for the value of direct work in these areas now, and in some cases is a larger effect than the chance your work comes to fruition after transformative AI.

Overall, I think that longer term projects do get down-weighted by these considerations, but their advantages sometimes outweigh that — especially if they are shooting for a very big payoff. I’d guess that if someone looked at their options and thought the best option was one that took 5 to 10 years to pay off, then about half the time it would remain their best option even after taking AI timelines into consideration. After all, it is not uncommon for your best option to be several times better than your second best.

So I think the community of people working on transformative AI are likely underrating types of work that need five or more years in order to pay off. The ideal portfolio of activities aimed at making the AI transition go well should include a number of things that really help us succeed in worlds where we get longer to try.

But I want to stress that none of this implies we can slack off.

We’re in a race against AI timelines. It is just that we don’t know if that race is a sprint or a marathon. In either case, time is of the essence.

Conclusions

We have seen that there is substantial disagreement and uncertainty about when AI will start having transformative impacts on the world. This is because there just isn’t enough evidence to pin it down. My claim is that for the purposes of planning we should adopt neither short nor long timelines, but broad timelines:

The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines.

Given this deep uncertainty we need to act with epistemic humility. We have to take seriously the possibility it will come soon and hedge against that. But we also have to take seriously the possibility that it comes late and take advantage of the opportunities that would afford us. The world at large is doing too little of the former, but those of us who care most about making the AI transition go well might be doing too little of the latter.

We need to take more seriously the possibility that the world will look very different at that time, which should broaden our own Overton windows about what kinds of plans could succeed. And we shouldn’t be ruling out all actions which take a long time to pay off. Even if they wouldn’t help in short timelines worlds, some actions more than make up for this with substantial impacts if timelines are long.

Funders, career advisors, and movement builders should be thinking about this with regards to how we act as a community: to the shape of the whole portfolio of work aimed at effectively improving the world. And each of us should be reflecting on what this deeply uncertain timing means for planning our own contributions over the years to come.