This is a personal guest post by Bentham’s Bulldog, created while they were a visiting scholar at Forethought.
In this piece, I’ll analyze whether it will be possible to get AIs to do good philosophy, as well as the other kind of work needed to plan for a good future. The biggest part of the challenge is that the answers to philosophical questions are generally unverifiable, so it’s less clear how AIs might come to know them. My aim in the piece will be to describe reasons to think AI for philosophy could be good, as well as to lay out concrete scenarios for how this might work.
I do not claim that it is a guarantee that we can get AIs that solve all the philosophical questions. My core claims are as follows. First, it is reasonably likely, though not guaranteed, that the world could in principle build AIs that discover the right answers to important moral questions (maybe 60% odds). I don’t know when this would happen—I’m just discussing the in-principle possibility. Second, even if AIs don’t get the right answers to important moral questions, it is likely that they will still be closer to the right answers than humans. While I suggested in the last piece that the default scenario makes it very unlikely that humans will act in the morally optimal way, or anything close to it, I think the odds are better for philosophically reflective AIs.
The piece has the following sections:
How good might AI philosophy be? In this section, I’ll give three reasons for optimism: 1) philosophy is a priori, so it doesn’t require interface with the physical world and can plausibly proceed quickly; 2) in a world of superintelligence, AIs can create clever schemes to get highly philosophically competent AI; 3) presumably humans know all sorts of things about philosophy—whatever allows us to know such things can plausibly be mimicked.
GPT10 and Claude 8. In this section, I’ll describe one scenario where we get AIs that can make competent philosophical decisions—where we just get something broadly like current AI models but upgraded. This wouldn’t automatically solve every philosophical question, so it probably leaves lots of value on the table, but it would leave decisions in the hands of a wise and reflective decision-maker.
Philosophical competence by default. Another possibility is that as we build superintelligence, it will be philosophically competent by default. Perhaps the cognitive processes behind superintelligence allow one to figure out the answers to philosophical questions by default. To speed this up, we ought to improve AI for philosophy—train AIs out of philosophically sloppy answers and design philosophy evals to improve their philosophical ability.
The escalating ladder plan. In this section, I present a specific proposal for getting AI to do good philosophy. Specifically, we can have philosophers evaluate AIs for their philosophical competence, thus creating some AIs that are better at philosophy than current people. Those AIs can evaluate the philosophical acumen of the next generation of AIs, who evaluate the acumen of the next generation, and so on. This can work insofar as one can, in general, assess the philosophical competence of those more competent than oneself.
The Carl plan. In this section, I give a proposal from Carl Shulman where we build a bunch of different superintelligences that use different belief-forming processes to get at the answers to difficult questions. We see which one is best at figuring out the answers to verifiable questions, and then ask it to answer philosophical questions that aren’t verifiable. The core idea is that whichever processes are best for figuring out the answers in verifiable domains are also likely to be best for figuring out unverifiable domains.
Conclusion. In the last section, I conclude, and discuss the prospects for combining a number of different above proposals.
How good might AI philosophy be?
In the last piece, I argued that one of the big reasons to hand off important decisions to AI is that AI might be better at philosophy than humans—less prone to make the sorts of errors that could jeopardize nearly all future value. But you might wonder: how good at philosophy will AI really be? Would AI really be able to discover the moral facts if there are any, and if not work out a compromise among reasonable moral theories? In later sections, I’ll discuss specific proposals and scenarios for AIs being good at philosophy. Here I’ll provide some fairly general considerations for it being possible to make AI philosophically competent.
A first consideration is that philosophy is a mostly a priori domain. To figure out the right answer in Newcomb’s problem, you don’t need to do any physical experiments. At various points in the intelligence explosion, we should expect crucial bottlenecks to concern interface with the physical world, after there are extremely large numbers of AIs capable of performing adroit cognitive feats. This is a reason why we should expect a lot of the philosophical progress that occurs to go more quickly than progress in narrowly empirical domains. This obviously doesn’t suffice to show that AIs will be good at philosophy, but it’s a reason to expect potential progress to be quick.
A second consideration is an analogy with humans. Humans know all sorts of things that aren’t directly verifiable. For instance, I take myself to know each of the following:
Stars that have receded past the point of the visible universe continue existing.
A leprechaun did not fizz into existence spontaneously in my bedroom two minutes ago, make himself a cup of soup, and then disappear.
The universe is billions of years old, instead of five minutes old and created with the appearance of age.
It’s not just simple things. I think people know the answers to all sorts of difficult philosophical questions—though there’s obviously disagreement on exactly which one (I claim to know, for instance, that thirding is the right answer in Sleeping Beauty, though feel free to plug in some other example of apparent knowledge if you disagree). Similarly, plausibly philosophers of today know that logical positivism is false even though historically it was believed by a number of serious philosophers, that average utilitarianism isn’t the right moral view, that the external world is real, and a number of other non-obvious things. Evolution did not specifically design us to have philosophical knowledge—it designed us to reproduce, made us smart and built into us a bunch of intuitions as a means towards maximizing our inclusive genetic fitness, and philosophical ability was the eventual result.
Plausibly we, however, will be more directly optimizing for something in the vicinity of AI philosophical competence. We want to build AIs that can think clearly about important subjects. Thus, one should expect AIs, in the limit, to be more philosophically competent than humans. Since humans are already capable of reasonable philosophical competence, we should expect AIs to be very philosophically competent.
Now, you might object that the answers to many of these questions are in some sense constituted by facts about us. If, say, the moral facts are facts about what we’d value under certain idealized conditions, then it’s no surprise that we have some insight into them. But this response is a lot less plausible as an answer to how we possess certain bits of knowledge in non-verifiable domains. Whether the A-theory or B-theory of time is true or whether God exists isn’t constituted by any facts about our attitudes. Furthermore, if the truth in some domain is constituted by facts about our attitudes, that would be easier for AI to ascertain because what we’d care about in certain counterfactual settings is verifiable. Lastly, presumably whether the moral facts are exhausted by facts about our idealized attitudes is not itself verifiable—so if one takes oneself to know that the moral facts reduce to the attitudes of ideal observers, then they must still think humans have knowledge about some unverifiable domains.
Another possible concern is that perhaps the faculties that evolution furnished us with that let us solve philosophical problems will be closed off to the AI. For example, perhaps there are some philosophical problems that you need to be conscious to solve (e.g. you might need to be conscious to have adequate understanding of the nature of consciousness in order to learn facts about it). So then if AI isn’t conscious, it might not know these things. An AI that lacks the ability to experience pleasure or pain might lack the ability to see the desirability of pleasure and the undesirability of pain.
This is some worry, but two things mitigate it. The first is that AIs, being trained on human data, will probably pick up a lot of the intuitions that humans have. Current AI models agree that pointless intense pain is a bad thing. It is similarly plausible that the AIs of the future will be able to, in some sense, mirror the judgments of reflective humans on these matters.
The second is that if this is right, we should create conscious AIs to do moral reflection. If one needs consciousness to figure out the answers to tough philosophical questions, we should have conscious AIs figure out the answers to tough philosophical questions. Even views that hold that consciousness is substrate dependent normally hold that digital minds of the right kind would be able to be conscious.
This argument from analogy isn’t completely dispositive. There are a number of important differences between humans and AIs (though some of these make it more likely AIs would be able to form true beliefs on non-verifiable subjects). But it’s at least some reason to think AIs doing philosophy is a realistic possibility.
The third and final consideration favoring the possibility of AIs for philosophy is that in a world of advanced AIs, we’ll have a very large amount of cognitive labor that could be used to design increasingly clever schemes for AI philosophy. So even if we can’t currently think of a regime that would enable AIs to reliably get the answers in unverifiable domains, we should think that a world with superintelligence is likely to uncover such a regime.
This isn’t a guarantee. This regime might be impossible in principle. Alternatively, to design such a regime, one might already need to be able to find the answers to questions that aren’t verifiable in principle. But at the very least, it gives some reason for optimism.
GPT10 and Claude 8
Current AI models can give reasonable answers to philosophical questions. The answers they give strike me as somewhere below the level of Ph.D. students or professional philosophers, but above the level of most undergraduates. In the future, AI will probably become better at this. In such a scenario, they’d develop no new qualitative ability to solve every philosophical question, but instead merely begin to resemble extremely sharp philosophers—ones more able to formulate objections, precisely distill claims, and so on, than the best philosophers.
This seems like a lower bound on how good AI philosophy could get. It would be surprising if there was any in-principle impossibility in designing AIs to do philosophy better than current human philosophers. Perhaps it won’t be able to find the answers to all the world’s questions, but at the very least, it seems like it will be able to somewhat exceed the best human philosophers.
Is this far enough? Probably we still lose out on most expected future value in this scenario. To access most future value, one must, as previously discussed, answer a number of very difficult philosophical questions correctly. Philosophers are very far from converging on the answers to these questions; AIs are likely to be as well.
But still, this should be enough for us to get generally sensible, level-headed AI decision-makers who are morally motivated and carefully consider philosophical questions. That’s not a perfect scenario, but surely it is better than the default one—of widespread AI disenfranchisement and poor decision-making.
Philosophical competence by default
A more promising possibility—perhaps the process of building superintelligence gets philosophical competence by default. This can be supported by the earlier evolution analogy: there was no direct selection for true philosophical beliefs (it isn’t as if those who got the right answer in Newcomb’s problem were likelier to pass on their genes). Nonetheless, evolution selected for a broader faculty of reason that enabled the discovery of philosophical truths.
It may be that to be superintelligent, one must develop a kind of general intelligence. This general intelligence allows one to accurately discover the truth about philosophy, as about other domains. This isn’t anything like guaranteed, but it seems like a live possibility, in light of how much philosophical acumen AI already has. One reason that it’s not guaranteed is that reasoning might only push one towards greater coherence. It may be that by reflecting carefully, one’s beliefs become internally consistent—but insofar as one has wrong starting points, one might remain in error. It could be the other way; it might be that having intuitions that allow one to be superintelligent across verifiable domains also enable one to be superintelligent in non-verifiable domains. It could be that whatever skills are required to be highly accurate across domains give one sets of epistemic practices that generalize to non-empirical domains.
To my mind, it is not obvious if general superintelligence gets you superability in philosophy. I could see things going either way. But if it does, then the problem of getting philosophical superintelligence is fairly easy.
The escalating ladder plan
In general, one’s ability to recognize good philosophy surpasses their ability to perform good philosophy. It is easier to know that, say, Parfit is doing very good philosophy than to do philosophy at his level. If this generalizes, it might be possible to have an escalating ladder of philosophical competence, starting from human philosophers. This is similar to the method of scalable oversight.
Here’s how it works: human philosophers would be consulted to design evaluations for AIs. AIs could be graded by human philosophers on their philosophical competence, and modified so as to be more competent. From this process, in the limit, we should expect to get AIs that are somewhat more competent than most human philosophers—if human philosophers can assess philosophy above their level.
Then, we use those AIs to prompt the next line of AIs. Those AIs would assess the philosophical competence of the next round of AIs being trained, who would assess the competence of the next round, and so on. If at each level, one can assess those who are better at philosophy than they are, this could lead to an escalating spiral of greater and greater competence. Each round pushes competence to be greater, reaching the outer edge of where they can competently assess.
At each level, assessments would be designed based on philosophical competence, not agreement. The philosophers assessing the AIs, for instance, would assess how good their reasoning was and how much philosophical skill they display, rather than whether they think they got the right answers. Of course, sometimes getting sufficiently wrong answers is evidence for malignant reasoning (something must have gone badly wrong if you ended up concluding that the Earth was flat) but they should aim to keep things maximally theoretically neutral.
One could also train a number of different AIs to train the next generations of philosophers. One could look for convergence across different models. If they were getting at the truth, one should expect them to converge. If there was only convergence on some subjects and divergence on others, one could trust that they were getting at the truth on the matters where they were converging.
That this would work is not a guarantee. It could be that philosophy could be equally good in some formal sense while coming to a number of different conclusions. Perhaps, for instance, the ideal Kantian philosopher is neither better nor worse than the ideal utilitarian. Escalating competence might be liable to lead in a number of different directions, never converging on the truth. But this is far from obvious. Just as some level of competence in biology allows one to see the truth of the theory of evolution, presumably there’s some level of philosophical competence that allows one to see the truth of certain moral theories—or, if there aren’t moral truths in this sense, that allows one to make as much moral progress as can be made. The escalating ladder of evaluations might allow one to find this.
The view on which philosophy is about purely formal competence will also struggle to explain how humans have as much philosophical knowledge as we do. There are a number of subjects on which we have substantive knowledge, even though the alternative view is internally consistent and faces no strictly formal issues. Whatever one’s diagnosis of how this came to be also plausibly generalizes.
It could also be that as competence grows, eventually one reaches a point where their ability to recognize good philosophy does not surpass their ability to do it. Yet insofar as this pattern holds across other domains, and seems to hold in philosophy, it would be a bit surprising if it abruptly stopped at some point.
The Carl plan
Arguably the most promising plan for getting AI that can do philosophical reasoning was formulated by Carl Shulman. The idea is as follows. Start by training a number of different superintelligent AIs that obey different epistemic constitutions. These would involve different broad approaches to reasoning. They might differ with respect to:
The weight placed on intuitions.
The ranking of theoretical virtues.
How much they value matching the data vs simplicity.
Preference for different kinds of intuitions (whether about direct cases or about broader and more theoretical matters).
Approaches to generalizing from empirical evidence to non-empirical domains.
Then, see which of these constitutions does best on verifiable matters. These might be in math, the physical sciences, forecasting, and more. Then, ask those superintelligences for the answers to non-empirical questions and trust their answers. The idea behind the approach is that one should take the best epistemological practices from verifiable domains and then apply them to unverifiable domains. This is the best way to get good answers.
This might not work. It could be that the right ways of reasoning in verifiable domains differ from the right ways of reasoning in unverifiable domains. Perhaps, for instance, reasoning in verifiable domains is mostly about having parsimonious theories to fit the data. Having the right epistemic constitutions might not suffice to get the right answer; being right about certain non-verifiable questions might come down to having the right starting sets of intuitions. In morality, for example, it may be that reasoning alone won’t get you the right ultimate view unless you have the right intuition. But this proposal seems to get us about as close as we can get to getting the right answers. If we cannot get AIs good at philosophy by having them have superintelligent and accurate constitutions in other domains, it is hard to see how we could be assured they’ve gotten the right answers.
In addition, even if getting the right answers requires having the right starting intuitions, techniques for training superintelligence might produce generally accurate initial intuitions. Human intuitions often go wrong because of various biases. We often neglect the scale of a problem because we can’t visualize it accurately. AIs could be different. Presumably superintelligence, in the limit, wouldn’t display biases of the standard sorts. So this source of common error in intuition would be absent.
One could also take a pluralistic mix of the views of the superintelligences with the best constitutions. That way, there is less chance for the ultimate plan for the universe being ruined by the idiosyncrasies of any particular epistemic constitution. Wisdom of the superintelligent crowd might root out the remaining errors, where their commonalities might be centered on the truth.
Conclusion
You might worry about handing off big decisions to AI if you’re skeptical that AI can do good philosophy. Here I’ve given some reasons to think AI might be good at philosophy and discussed more specific proposals for ensuring such a thing. Overall, it’s far from a guarantee that AI can be arbitrarily good at philosophy, but likely they can be much better than human decision-makers. Importantly, one could employ a variety of these methods, and then trust them in the areas where they converge. If these very different approaches turned out similar answers, that would be evidence that they were tracking the truth.
This article was created by Forethought. See all our research on our website.
Thanks, this changed my mind a little bit
I predict that ASI, demonstrably far smarter than any human, will tell us that moral realism is false and yet philosophers such as the author will still refuse to accept it.