James Tillman is a research fellow at Forethought, and Tom Davidson is a senior research fellow at Forethought. They discuss whether AI could rapidly make itself smarter just by improving its own algorithms, without any new chips or data centres (a "software intelligence explosion"), and what might stop it. They cover:
What a software intelligence explosion is, and why it would be faster and harder to govern than one that needs new chips and factories
Why the data that would settle these questions sits inside AI companies, and the case for making it public
Scale-dependent algorithms: why breakthroughs like the transformer and mixture of experts only pay off at large scale, and what that means for automated AI researchers with small experiment budgets
How far it could go: how much more efficient than the human brain AI could become, and why James doubts the estimate that rests on Chinchilla scaling (an ideal brain the size of a honeybee's, learning for a million years)
Whether data bottlenecks, or AI that excels in the lab but not in the real world, could stop it, and why Tom thinks they wouldn't
Further reading: “How quick and big would a software intelligence explosion be?” by Tom Davidson and Tom Houlden, and Tom's “Data Bottlenecks Won't Stop an Intelligence Explosion”.
Chapters
00:00 What is a software intelligence explosion?
02:49 The data AI companies have and we don't
05:02 Compute versus algorithms
07:21 What are scale-dependent algorithms?
11:00 Could AI researchers find new algorithms with little compute?
19:23 Are estimates of algorithmic progress inflated?
26:08 How far could it go? Catching up with the human brain
34:39 Beating the human brain: Chinchilla scaling and a bee-sized brain
49:01 Could data bottlenecks stop it?
50:48 Would AI trained in the lab work in the real world?
58:01 Could we run out of data first?
1:02:35 How big a speed bump is data?
1:07:20 The verdict
Here’s a link to the full transcript.
ForeCast is Forethought’s interview podcast. You can see all our episodes here.


