In 2018, the long reflection was long. Will MacAskill, introducing the idea: "maybe it's 10 billion people, debating and working on these issues for 10,000 years because the importance is just so great." Or, on another podcast that year: "Perhaps it's a period of a million years or something. We've got a lot of time on our hands.”
By 2026, it seemed to have gotten short. From the description of viatopia: "given the sheer scale of cognitive abundance that superintelligence could unlock, the reflective process might not need to last very long in calendar time.” My impression is that thinkers in the field take seriously the idea that it might just take weeks or months.1
This change seems to be driven by assumptions about how much AI systems would be involved. The thought goes: if AI compresses a century of intellectual progress into a decade, it compresses reflection along with everything else.
I don’t think that’s quite right because it conflates different notions of “reflection”, which run under different constraints in terms of how much AI can speed them up.
Two different types of “reflection”
Learning: coming to more accurate beliefs
This covers all domains where there is a fact of the matter: e.g., empirical and mathematical facts.2 Superintelligent AI systems can help enormously in these areas. They can generate new arguments, evidence, and considerations in parallel and at incredible speed.3 So AI systems will quickly end up with much more accurate world models compared to humans.
However, this does not automatically translate into more informed decisions because decision-makers would need to act on those improved models of the world. If those decision-makers are the AI systems themselves, that’s trivially true. If humans keep a role in decision-making, they need to decide how much to defer. Wholesale deferral could happen instantly; understanding and digesting the new knowledge would lag behind and proceed on more ordinary human timescales (but still be meaningfully sped up compared to the default).
It seems likely that substantial, or even wholesale deferral to much more accurate AI systems would quickly become the norm because those systems would repeatedly be proven right and selection effects would favor those who defer quickly. Such a regime is not even that unusual: deferral to epistemic authorities is already the norm in most societies right now.4
Becoming: changing what you value and how you decide
This is arguably the real prize of the long reflection or viatopia: becoming wiser and realizing what we truly value.5 Under most plausible views, at least some aspects of these questions are not facts to be discovered.6 Instead, they require us to make decisions without any privileged procedure for making them. As Joe Carlsmith has argued, it’s choices all the way down.
In this process, the role of AI systems is much more limited. It’s true that they can scan my brain and run a gazillion simulations of me to tell me what a future wiser version of me would do in my place. However, that result is not in fact determinative of what I value. For one, it’s unclear what reflection or idealization procedure to use because there is no single privileged one: which one to trust if they output different results?7 More damningly, however, no output constitutes an answer. Concretely, suppose the AI system told me any of the following after running all these simulations:
“If only you reflected more, you would come to value the maximization of this particular crystalline structure.”8
“If only you meditated more on the nature of existence, you would come to see the world as perfect just as it is without any need to change it.”
“If only you became a more whole person, you would realize that your thirst for revenge doesn’t actually serve you and is getting in the way of living a good life.”
“If only you thought more deeply, you would come to see the moral truth, which requires you to instruct me to replace all you hold dear in this world with supercomputers that instantiate quadrillions of super happy beings.”
From my current vantage point, I do not in fact value that particular crystalline structure, do not see the world as perfect, have not given up my desire for revenge, and have not been moved by the moral truth. The change in values is not produced by being told about the final destination. It is only ever produced by the journey of getting there, which I have not yet undergone.9 Anything any AI system can tell me is only an input into my own choices about what I value and who I want to become. I would still have to do the deciding and the becoming.10 There is no shortcut available that doesn’t run through actual engagement, experience, and practice, which runs on the timescale of a human life.11
Even on moral realist views, there is a further question whether I am in fact motivated to act on the moral facts once I learn about them — whether I sufficiently value doing the right thing. On most accounts, this is not accomplished by merely being handed them. Instead, it requires either a deep and meaningful grasping of them or a cultivation of moral virtue to want to act in accordance with them. Both ask for our participation, which means there is a choice whether to participate or not. For instance, whatever moral truths the AI systems discover may seem alien or abhorrent to us at first, which would disincline us to follow their advice. We would need to want to overcome that, which we may simply not. That being said, AI systems could probably meaningfully speed up the process of discovery and learning of moral facts if they indeed exist and we are sufficiently interested in them.
Predictions about what I will in fact choose, e.g., “You will reflect and then choose X” or “You will hand off to me to do Y”, do not help either. Because they are an input into the choice they predict, they can only be accurate if they remain true after being heard. And a prediction that survives being heard is either telling me what I would have done anyway, or it is a self-fulfilling prophecy, which does not help me decide but decides in my stead.12 Neither is speeding up my own reflection in any meaningful sense.
I can of course simply hand off any decisions to an AI system. It can even inform me about the consequences of this handoff and what it would do on my behalf. But that just relocates the question because I still have to decide which AI system to hand off to: the one that would maximize the crystalline structure I would come to value (but don’t right now), the one that would act in accordance with the moral facts (even if I don’t like them), or the one that would implement my current values. The AI can’t tell me which is right. It can tell me what I would do if I reflected more, but it leaves us with the same question we started with.
Ultimately, the role of AI systems in this process of becoming is not that of an oracle or a scientist. To participate in becoming is to be a mentor, coach, friend, or artist who shapes my choice through witnessing, encouraging, inspiring, teaching, and warning. Just as with humans in these roles, we will have to learn to discern whether someone’s influence is serving or undermining us.13
Length is a choice
The Bootstrapping to Viatopia post already states the options: perhaps deliberation needs "thousands of years (because human decision-makers want to go through the full reflective process themselves, and aren't willing to defer to AI advisors). Or perhaps we could be done over the course of just months."
We can only get to "months" by handing off almost all “reflection” to AI systems. For learning, we will have good reason to delegate, at least at the point where they have proven themselves to be epistemic authorities. It simply helps us achieve our aims better. For becoming, "deferral" is not a harmless delegation of “reflection”. Instead, we would effectively hand over all of our power and authorship over the entire future to our future selves. That may sound harmless or even desirable, and they may seem wiser and more moral to us in some ways. But I expect that, after thousands of years of reflection, they will also seem extremely alien to us. They will want to create worlds we don’t understand or care about because we are separated by thousands of years of reflection we never actually underwent. In that scenario, it’s not obvious to me that simply deferring or handing off control is the right choice because I would still have to live in that world, with the values I have now.
This article was created by Forethought. See all our research on our website.
See, e.g., footnote 4 of the Long Reflection Reading List.
To what extent philosophical questions, including moral ones, are included here is a matter of philosophical debate. Realists of various flavors would maintain that there is a fact of the matter whereas anti-realists would deny it.
There may be some questions where their cognition may still quickly hit limits because there is just not much additional evidence that can be generated or discovered, e.g., whether we live in a simulation or how the universe was created.
The main reason people might not defer is that the systems will likely also discover many “uncomfortable” truths that some groups would rather not believe. Carl Shulman has discussed this dynamic on this podcast.
I mean to include all normative matters like values, decision theory, epistemology, and meta-philosophy.
There is a narrow case where this is covered by “learning” alone. It would require something like moral/normative realism combined with motivational internalism, where grasping those facts is itself overwhelmingly motivating. For all other positions, including even weak internalism, there is at least some residue of “becoming” that’s required.
If the outputs are commensurable, one might hope to aggregate across procedures, as the moral uncertainty literature does across theories (e.g., maximizing expected choiceworthiness). But the choice of aggregation rule is one more level of the same problem. And even if simulations of me resolving every level of this uncertainty converged in the end, that would only tell me what I would eventually conclude, given certain kinds of reflection. It doesn’t tell me whether that “I” is in fact authoritative for me right now.
Adapted from Joe Carlsmith.
This is a kind of transformative choice in L.A. Paul's sense. AI simulations do not dissolve the fundamental problem she discusses: everything an AI can hand me is relayed knowledge about the far side. Possessing the knowledge itself would just mean having been transformed; the information and the transformation are one thing, and that thing cannot be handed over as declarative knowledge.
Another intuition pump runs through personal identity: the beings coming out of the other end of the simulation are not me. They are someone else. Whether I ask them to make decisions for me is akin to asking a friend to make all decisions for me. They may seem wiser to me, but they are still not me. So a meaningful choice remains.
It arguably even runs on the timescale of human generations.
There are three possibilities for an AI trying to predict what I will do: (1) There is exactly one prediction that remains true after I’ve heard it. Then either hearing the prediction makes no difference because I would have acted this way anyway (which means the prediction does not “help” me make a decision in any meaningful sense), or the prediction itself causes its own truth, which means it is exerting non-reflective influence over me. (2) There is no prediction that holds true after being given, e.g., because I’m defiant of any prediction given to me. Then there is no true thing the oracle can say to me about my choice at all. (3) There are multiple predictions that would each become true upon my hearing them (e.g., told I'll grow cynical, I would; told I'll grow generous, I would). In picking which one to reveal, the AI is choosing my choice rather than revealing it. If I tell the oracle how to choose, I am back to choosing myself. How do I do that? Do I reflect before choosing?
To discern, we cannot judge by outcomes and will have to judge by means and quality of relationship. In the case of AI systems, we may even be able to audit some of these qualities, but then the discernment will just have to move to the level of the audit.
